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(54) Methods for creating an image for a tiiree-dimensionai display, for calculating deptli 
information, and for image processing using the depth information 



(57) A method is proposed for automatically obtain- 
ing depth information from a 2-D motion image, so as to 
create an image for a 3-D display. Further, methods are 
proposed for selecting appropriate frames for the calcu- 
lation of the depth infonnation, or discontinuing the cal- 
culation, and for conducting image processing using the 
depth infonnation. Examples of various types of image 
processing can be listed as including the crea^on o' a 
viewfinder image seen from a different point, natarai 
scaling operations to an image area, and separation of 
a desired image area. First, a motion information of an 
object on a screen is extracted by block matching or the 
like. Second, the actual movement of the object in the 
3-D world is calculated. Since the viewfinder image is a 
projection of a 3-D space, it is possiole to obtain the orig- 
inal 3-D movement of the object, based on the move- 
ments of a plurality of representative points through an 
inverse transformation, the representative points being 
provided in the viewfinder image. Resultantly, 3-D coor- 
dinates of the object are identified, so that depth infor- 
mation of the object is obtained, Aftenwards, a parallax 
is calculated based on the depth information so as to 
create right and left eye images from the input viswtinaer 
Image. Alternatively image processing, such as sepa- 



ration of an object having a depth within a predeter- 
mined range, is carried out based on the depth informa- 
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BACKGROUND OF THE INVENTION 
* Pteld Of the Invention: 

[0001] The presenl invention relates to nelhods Icr creating an Image for a three-dimensional display (hereinafter 
referred to as a 3-D display), for calculating depth information, and for image processing using the depth information. 
The method for creating an image for a 3-D display oarticuldri/ teldtes to a method for creating a pseudo-viewfinder 
« image shot by a multi-eye camera (a stereo imaqe) (torn a viewfinder imaqe shot by a monocular camera. The method 
for calculating depth information relates tc a notrcr fc ct c, ^ dunr o t {. w c i an object and a viowor which 
is applicable to practice the method for creatmq an maqc rcr .1 3 D disp ay : :ic niclnod for image processing using 
the depth information relates to applications including me creation of an image for a 3-D display and the suspending 
of creation of an image for a 3-D display, an enhanced display or the liKe. 

Description of the Prior Art: 

[1] Creation of an Image for a 3-D Display 

20 [0002] In fields related to television techniques for creating a 3-D image (a pseudo stereoscopic vision) through the 
detection of a movement of a 2-D motion image have been known. One typical example of such a technique is a 3-D 
display employing a time difference method, the principal of which will be described with reference to Figs. 1 to 3. 
[0003] In a scene where an object moves from left to right while the background stays still, as shown In Fig. 1 , by 
reproducing respective images for right ard left eyes (hereinafter respectively referred to as right and left eye images) 

2S so as to have a predetermined lapse of time between them as shoivn m 2, a parallax 6 is caused, as shown in 
Fig. 3. "A parallax" or "a binocular disparity" is defined as an angular difference betwt-en sight vectors directed at one 
point from right and left eyes, respectively'. In Fig. 1 . since a v ewer perceives the car as being closer than the .back- 
ground due to parallax, a pseudo stereoscopic vision is achioved. When the object, the oar in this case, moves in the 
opposite direction, respective images should be reproduced such that the one for a rignt eye is reproduced earlier than 

30 the one for a left eye by a predetermined time, contrary to the examDie shown in Fig 3. 

[0004] JP Publication No. Sho 55-36240 discloses a display apparatus to- a s:orocscopic image using depth infor- 
mation, in whtohthe apparatus receives only an image signal shot 'rom one bas c direc:tcn (that is. a 2-D motion image) 
among signals from multiple directions and a signai coniainng Gepih infoimation 0' an otjject, so as to reproduce 
within the apparatus the original viewfinder image shot irom mjitipie direct ons, Tht purpose of the apparatus Is to 

35 reduce a transmission bandwidth. The apparatus incorporates a variable delay circuit for causing a time delay while 
controlling the extent thereof according to depth information. The time delay results In a parallax. According to an 
output signal from the circuit, image signals are reproduced for right and left eyes, in this way, a pseudo stereoscopte 
image is displayed. This publication discloses, as a preferred emboaiment of the disclosed apparatus. (1) a device for 
displaying a pseudo stereoscopic Image for a viewer by -especlvely supplying right and left eye images to two CRTs, 

*3 which are situated forming a predetermined angle with a half m:rror and (2) a device for displaying a pseudo stereo- 
scopic image for a viewer even if the viev/er moves in a horizontal airecticn, using a Isntlcular lens fixed to a display 
screen. 

[0005] However, the above apparatus works on the condition that depth information is supplied externally. Therefore, 
if it only receives a 2-D motion image the apparatus cannot create a pseudo stereoscopic image. 

« [0006] JP Application Laio-Open No Hei 7-5911 9 also discloses an aoparatus for creating a pseudo stereoscopic 
image based on a 2-D motion image. The apparatus cornp -ses a datectton crcuit for detecting a motion vector from 
a supplied 2-D motion image, and a delay circuit for doiaying :iiher a light or a ief! image according to the motion 
vector The delay causes a parallax. This application disciosss. as a preferred embodi.Tientof the disclosed apparatus, 
a head mounted display (HMD), which is a glasses type display 'or supplying different images :o right and left eyes, 

50 Through the HMD, a viewer can see a pseudo stereoscopic imsge. 

[0007] In this apparatus, however, since the extent of delay Is determined according to the magnitude of a motion 
vector, any object moving at highspeed appears to be closer tc '.le vewer resulting ir an unnaturaf stereoscopic view, 
which is discordant to the effective distance between the viewer -o' the camera) ana the object (that is a depth), 
[0008] JP Laid-Open Application No She 60-263594 -liso diuclcses an apparatus for displaying a pseudo siereo- 

55 scopic image using a time difference me-hod, m which the apparatus displays right and left images alternatively for 
every field, so that they are seen alternatively via snutter glasses for every field, as the shutter glasses alternatively 
open their right and left eyes. This application further discloses a method for generating a stereoscopic effect by pro- 
viding a longer time difference between right and left images when an object moves at low speed. However, since this 
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apparatus also does not operate based on depth information, it is not really possible for an accurate pseudo stereo- 
scopic image (o be created and thus displayed. 

[00091 "PIXEL" (No. 128), a magazine, issued on May 1 . 1993 describes in pages 97 to 102 a pseudo stereoscooic 
image systenn using depth information. In the system, an obieci is first displayed as a grayscale image where the gray- 
scale level corresponds to the depth, and then based on the gray level, the appropriate parallax is calculated in temis 
rlllT f leflimagesaroc.«ated-.obe«eenviashuuor glasses. However, the perspective 

image is manually created and a technique for automating the creation is not disclosed 

[0010] National Publication No Hei 4-504333 (WO88/04804) discloses a methodfor achieving a pseudo stereoscopic 
lethoc comprises steps of dividing a 2-0 motion image into some areas, forgiving 
so as to provide sach of the areas with a parallax, and tor creating a pseudo 
stereoscope image. However, the depth information is manu...lly supplied and a technique for automating the supply 
IS not disclosed. ' 
mm In a research field called "Computer Vision," a study has been conducted mto a method for estimating a 3-D 
structure and movement of an object, Concretely speaking, the study, which is aimed at self-control of a robot, relates 
to acquisition of an accurate distance from a viewpoint to an object by either shooting the object using a stemo camera 
(a multi-eye camera), or using a monocular camera while moving it. Several aspects of this technology are described 
in a report, entitled "1990 Picture Coding Symposium of .Japan (PCSJ90) " for example, on page 57. 

[2] Creation of Depth Infonnation 

[0012] Computer Vision would enable delect on of the depth of an object However, in the calculation of depth infor- 
mation, which IS based on 2-D motion infonnation, suitable images are not always supplied for the calculation. If the 
^Tfn^tTn'^ continued even with unsuitable images supplied serious errors are likely to be caused. That is, if depth 
infonnation is obtained from such unsuitable images, and then used tor the creation of a stereoscopic image it may 
be quite likely that the thus created stereoscopic image wil' be unnatural, i.e. exhibiting such anomalies as a person 
in the distance appearing closer than a person who actually is closer 

[0013] It is to be noted that the idea of obtaining depth information through understanding of a corresponding rela- 
'°noc..t^.!.? ^'^^^ ^"^^ For example, JP Application Laid-Open No, Hei 7-71 940 {which corresponds 

to USP5475422) mentions, as a prior art. (1) a technique for relating a point or a line between two images shot by a 
stereo camera, so as to estimate the position of the point or line in actual space (the 3-D world), and (2) a technique 
for shooting an object on a camera while moving it so as to obtain Its sequential viewflnder Images for tracing the 
movements of a characteristic point on the sequential viewfinder images, and thereby estimating the position of each 
characteristic point in actual space. 

[3] An Image Processing Method Using Depth Information 



[0014] A method for controlling the movement of a robot using depth information Is known, such as the foregoing 
Computer Vision. A method for creating a pseudo stereoscopic image using depth infomiation is also known such as 
IS disclosed in the foregoing JP Publication No. Sho 55-36240. 

[0015] On the other hand, a method for using depth infomiation in imago processing other than the creation of a 
pseudo stereoscopic image has scarcely been proposed. 

SUMMARY OF THE INVENTION 

[0016] The first object of the pi 



P'^esent invention relates to the creation or an image for a 3-D display, as described In 
he above (U In defining the object of the present invention, the inventor d.aws attention to the fact the all of the 
foregoing techniques forcreating a pseudo stereoscopic image have at least one of the following problems to be solved: 

1 , An accurate stereoscopic image based on depth infonnation is not created. Instead a mere 3-D effect is oro- 
visionally created according to the extent of movement. Further, since a parallax needs to be created using a delay 
in time (a time difference), horizontal movement of an object is required as a premiss of the creation which should 
constitutes a fundamental restriction. 

2. As it is not automated, the process for obtaining depth infomiation from a 2-D motion image requires an editing 
process. Thus, a stereoscopic image cannot be outout in real time upon input of the 2-D motion image. 

[0017] Therefore^ the first object of the present invention is to create an accurate stereoscopic image based on 
depth infonnation, by applying the foregoing technique relating to a computer vision to an image processing fieW in- 
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eluding technical fields related fo television 

[00181 In order to achieve this object, aacordinci to the Drcsent invsntion. deoth information is extracted from a 2-D 
motion image, based on which an image tc.r a 3-D display is created- i his is aoplymg a technique related to a computer 
visiontoatechnicalfieldrelatingtoanBTiacea^nld) -^c o ar^ -^saspprtolthp presentinvention, deptt-^ information 
Is obtained through the following processes, that s he -^ov ement of a 2-D motion linage Is detected; a relative 3-D 
nl between the scene and the shooting viewpoml ol ihe 2 0 motion image is cslculaled; and relative distances 
ing viow/poinl to the respective image parts ofwe i-D motion smage are oalculaled, based on the relative 
e respeaive image parts. Based on the thts obtained depth Intoirnation, a 
pseudo stereoscopic image is created 

[0019J This aspect of the present inven;ion can De differentt/ dascnbed as a depth being obtained through the fol- 
lowing processos: that is, a plurality of y ^ M ndo imc h - ifcrioierrLdtoasiramos)arosoloctedfroma2-D 



motion image ro be processed; a 



jf tho respective n 



parts is identified based on a 2-D positior al displacement between the frames. In other words, in order to dete.'mine 
the depth, 3-D movements of the respeciive image parts are calculated based on tie 2-D positional displacement, 
based on which positional coordinates of :he respective image parts in the 3-D world are calculated, according to the 
principle of triangulation. A frame is a unit for image processing, thai is. a concept including a frame picture or a field 
picture in MPEG, and the like. 

[0020] Regarding a 2-D motion image, a plurality of viewflnder frames are hereinafter referred to as "differeni-time 
frames," as they are shot at different times. {In the following description of a multi-eye camera, a plurality of frames 
which are simultaneously shot are referred to as "same-tim© f-amcs.") A positional displacement on a frame plane is 
referred to as "a 2-D positional displacement." In this aspect of the present invention, where different-time frames are 
discussed, "a 2-0 positional displacement" means a change caused along with a lapse of time, that Is, a movement. 
(On the contrary, "a 2-D positional displacement" of same-time frames means a positional difference among a plurality 
of frames.) 

[0021] The second object of the present inve-iiion elates lo .he calculation of depth information, as described in the 
above [2). That is, the second object of tne prcseni invention is to propose a method for obtaining a correct corre- 
sponding relationship among a plurality of images, so as to calculate accurate depth information, for selecting an image 
to be input appropriate for the calculation and for discontinuing the calculation of depth information when any iicon- 
venience occurs, such as could cause an unnatural pseudo stereoscopic image to be created. Further, the present 
Invention aims to propose methods for efftsctively determining corresponding and chaiacteristK points, and for search- 
ing and tracing points with a high accuracy 

[0022] In order to achieve this object, accotding to ;ho pieseni invention, two frames with appropriately large move- 
ments between them are selected from a motion image 3c tnat oepth Infoimation Is obtained from the two frames. 
According to this aspect of the Invention, it is possible to obtain a good calculation result, with pre-selection of frames 
which may facilllale the calculation. A judgement as to whethet frames have appropriately large movement or not may 
be based on the extent or variance of movement of a charactenstic ooint. 

[0023] According to another aspect of tne invention, with a representative point provided in a reference frame, the 
similarity of Images is evaluated between a image area including a characteristic point in the other frame (hereinafter 
refen-ed to as an object frame), and an image area ncluding t.^e representative poinr in the reference frame. A char- 
acteristic point is a candidate for a corresponding point subjcci lo an evaluation, the candidate being arbitrarily deter- 
mined. Then, the relative positional acceptability between the cftaracteristic point ana the other characteristic point Is 
evaluated. That Is, a judgement is made as to whether the relative positional relationship between the characteristic 
point and the other characteristic point Is reasonable or acceptable with respect to being determined as the same as 
the relative positional relationship between the representative point and the other representative point, respectively 
corresponding to the characteristic points When both evalu.qtions result in a favorab-e score, the characteristic point 
is tentatively determined as a correspond;ng point oi the '-epresenta-ive point. Subsequently, a best point is searched 
for where each of the evaluations yield tne best re.sult, by -novina one cor-esponding point within a predetermined 
search area, while assuming that all the other corresponding points are fixed (this method hereinafter being referred 
to as a fixed searching nnethod). The besi position, whicn has been found during the search, is detennined as a new 
position of the corresponding point. All cor esponding points are sequentially subjected to this search and the positional 
change. Afterwards, depth information is obtained Dased on e Dositionai relationship between a representative point 
and its corresponding point, the correspc^nding pent hsvin j been obtainoc Through the above mentioned series of 
processes. 

[0024] Conventionally, the similarity of maces has been aviSjaiez by Diock matct.mg or the lilce. In this invention 
on the other hand, by including an additicnal evaluation on the elatwe positional evaluation, the corresponding rela- 
tionship between frames can be more accurately detected. Th,i accuracy can be further improved through iterative 
calculations. 

[0025] According to one aspect of the present invention, the similarity of the image.s is evaluated by block matching 
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which IS modified such that the similarity is correctly evaluated to oe highest when the blocks including the identical 
object are tested, regardless of shooting conditions (h ^reinafier referred to as biased block matching) As to same- 
time frames, a certain color deflection tends to occur due to cnaractenslic d.fierences of a plurality of cameras As to 
different-time frames, the same problem will arise cue to changing weather from time to time as this causes a change 
in brightness of a viewfinder image. After correction is mace to solve such problems, the similarity of Images is trans- 
formed to ba expressed in the from of a geometrical distance, which is aconcept for judging the acceptatjiltty of relative 
positions. Then, the relative positional acceptability and the transformed similarity are combined together to be used 
for a general judgement on the evaluation results In :his case biased block matching may be conducted within a 
correction limitation, which is pre-determined depending or a distance tjetween the reference and object frames. That 
IS, when the distance is larger, a larger con-ection iimitatior; is set aKordingly 

[0026] A correction for off-setting a change in brightness is aisckjsod in JP Laid-Open No. Hei 5-3086630 However 
tne correction is applicable only to cases of facing-out or facing-in (a consistent changing in brightness) but not to a 
partial changing in brightness. 

[0027J According to another aspect of the invention, depth information is obtained through the following processes- 
tnat IS, a plurality of representative points are provided in a reference frame; a plurality of corresponding points of the 
representative points are determined ,n an object frame, so that each corresponds to a respective one of the repre- 
sentative points; and a positional relationship between at least a characteristic point among the representative points 
and its con-esponding points is obtained. As a characteristk: point, a point whose position moves steadily among a 
plurality of different-time frames is selected, because such a point is considered to be accurately traced 
[0028] Likeviflse. according to another aspect of the present invention, if a point, whose displacement between same- 
time frames is substantially consistent or changes substantially consistently, also shows similarly consistent movement 
or change in movement behveen other same-time frames shot at a close but different time from the above such a 
point may be selected as a characteristk; point. 

[0029] According to a further aspect of the present invention, depth information is obtained from the following proc- 
esses; that is, a plurality of representative points are provided in a reference image; a plurality of corresponding points 
of the representative points are detemtined in the other image; and a positional relationship between the representative 
point and its corresponding point is obtained; depth information is calculated according to the positional relationship 
in which the cateulation of the depth infomiation is discontinued when an insufficient number of characteristic points 
are established among the representative points or the movements of the characteristic points are too small, because 
It IS then very unlikely that a positional relationship between viowfinder images will be obtained with a high accuracy. 
[0030] Two conceptualty different corresponding points oxist, th,.i is, a true corresponding point and a computed 
corresponding point. In principle, each representative point has a sole corresponding point, eliminating the possibility 
of the existence of any other corresponding points in any other positions. This idealistic sola corresponding point Is 
the true corresponding point. On the other hand, a con-esponding point determined through calculations for Image 
processing does not necessarily coincide with the true corresponding point. This is the computed corresponding point 
which may possibly exist in any positions other than that of the true corresponding point, and change its position 
arbitranly. The positksnal change may be resorted in a process for improving the accuracy of the corresponding point 
as described later. In this specification, the term "corresponding point" is used to include both true and computed 
corresponding points without a distinction between the two concepts, unless ,t is necessary to differentiate them 
[0031 ] According to a further aspect of the present invention, a depth of a 2-D image is obtained in whK:h when the 
depth of any point in a certain image is calculated as negative, the depth is recalculated while referring to the depth 
infomation of points close-by with a positive depth value That is, when a depth Is cateulated as negative that is 
probably because of unsuitable variables being used during the calculation. Therefore, such a negative depth should 
be corrected based on the depth of a close point. 

[0032] The third object of the present invention relates to the above [3], that is, utilization of depth information in 
image processing other than the creation of a pseudo stereoscopic image 

[0033] In order to achieve this object, according to the present invention, in creating a stereo image by giving a 
parallax to a 2-D image according to its depth information, the parallax is first changed so as to fall within a predeter- 
mined range, so that the stereo image wilt be created according to the changed depth infomiation. An excessively 
arge parallax would cause fatigue on a viewei^s eyes Cn the contrary, an oxcossively small parallax would invalidate 
the meaning of a parallax as data. Therefore, it is necessary to keep a parallax wrthin a desired range 
[0034] According to another aspect of the invent-on , ,n creating a stereo imsge by giving a parallax to a 2-D image 
according to its depth infomtatbn. the parallax originally determined according to the depth infomiation is set to be 
variable. Wrth this arrangement, upon an instruction by a viewer to change a parallax, for example it is possible to 
create and display a pseudo stereoscopic image which is agreeable to the preference of the viewer 
[0035] According to a further aspect of the invention, in creating a stereo image by giving a parallax to a 2-D image 
according to its depth infomnation and displaying the stereo image on a stereo image display apparatus, a process to 
be conDucted to the 2-D image sc as to cause the parallax is detemiined according to a display condition unique to 
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the stereo image display apparatus. Ths display ccnd:ficn is goveired by the si^e of a display screen of the display 
apparatus, and an assumed distance frorri the display screen ic a viowor. 

[0036] According to a further aspect of (he inventic n. in crcattrg a stereo image by giving a parallax tor every image 
part of a 2-D image according to its depth ctormaticr, ar unsve-, mage frame outline: caused by the given para;iax is 

5 corrected, fvlors particularly. In giving a pa allax, if .-i- image area shown at the nght end of the screen, for example, is 
displaced siighlly rightward, the image arsa resuitartty proiecls of! the original shape of [he image frame, ane ttius 
causes uneven parts along the edge of ths image frame. A correction made to such an uneven part would straighten 
the appearance of the frame. The correction may be maae by uniformly cutting off a Deripheral part of the frame at a 
certain width so as to achieve a desired f.hape of the image frame. 

»o [0037] According to a further aspect of .he inventisn, in a meihod where Image processing is carried out for a 2-D 
imago according to its depth infonnatlon, an image a'ea subject to the imago processing is dctemiined, based on the 
depth information. With this arrangement it is possible .o separate an object or to change the scale of an obioct a 
certain distance from a viewer. 

[0038] According to the further aspect of the invention, m a method where image processing is carried out on a 2-D 
« i -.-T'^ge according to its depth infomiation, images with viewpoints at a plurality of points on a hypothetical moving path, 
where a shooting point of the 2-D image is hypothetically moved, are created tor use as a slow motion image, based 
on the depth information, 

[0039] It Is to be noted that, according to the present invention, a vlewfinder image seen from a different point may 
be created according to depth information A positional displacement of respective image parts, which will be caused 
20 accompanying a change in the view point, are calculated based on depth information, so that a viewfinder image is re- 
created so as to correspond to the positional displacements causeo. When a viewpoint Is changed in height, for ex- 
ample, a displacement (the extent of translation and rotation rnovemenis) of the object (respective image parts) can 
be calculated based on the distance by which the camera nas moved and the depth information, so that a desired 
viewfinder image will be created based on the calculation result 



BRIEF DESCRIPTION OF THE DRAWING S 



Fig. 1 is a viewfinder image where an obiect moves from lert to -ight while a background stays still. 
Fig. 2 shows reproductions of right ar,a left images having a time lag between tham. 
Fig, 3 shows a parallax caused duo ki the lapse of time of Fig. 2. 

Fig, 4 shows main stages for the creation of an image for a 3-D display according to Embodiment 1 . 
Fig. 5 is a flowchart for the detection of a corresponding relationship between viewfinder frames. 
Fig. 6 shows representative points provided In a reference frame t. 
Fig. 7 shows block matching. 

Fig. 8 is a conceptual model where Ihe value of Et is indicated for a -lentative corresponding point Pt'(i,i) in a 
perpendicular direction. 

Fig. 9 shows a relationship between a representative point and its correspondinc) point determined at S12. 

Fig. 1 0 is an explanatory diagram regarding a principle of evaluation of a relative position of corresponding points. 

Fig. 11 shows a result of improvement processing conducted on candidates for con-esponding points in Fig 9. 

Fig. 12 shows a relationship between movennenis of Point P on a screen and a S-D space. 

Fig. 13 is an explanatory diagram regarding a principle of detennining 3-D coordinates of Point P, based on the 

3-D movement of a camera and the movement of Point P on a screen. 

Fig, 14 shows representative points each giver an actual n.;rne-ic value. 

Fig, 15 shows a parallax given accorciing -o decth in'orrngtion. 

Fig, IS shows nght and left images created from Frairie t. 

Fig, 17 shows a non-linear transformation with respect to a parallax. 

Fig. 18 shows an example of a hardware structure for p a^ticinp Embodiment 1 . 

Fig, 1 9 is a monochrome picture showing a vie>Mindsr image in Frame t. 

Fig, 20 is a monochrome picture showing a vie-wfinder image in Frame t'. 

Fig, 21 is a monochrome picture of Frtime ; overl?iid virin =, grjd f.o- division, and provided with representative points 

Fig, 22 is a monochrome picture showing iniiia scsr. o-is M co.Tesponding point;; in Frame t'. 

Fig, 23 is a monochrome picture sf-ovving coricspondmi^ points at rnproved positions in Frame t'. 

Fig, 24 is a monochrome picture embodying depth information with a gray-scale image. 

Fig. 25 is a monochrome picture of a right imags created -according to depth information. 

Fig. 26 Is a monochrome picture of a left image oreatec according to depth infonnation. 

Fig. 27 shows main stages for the cre-ation of an image for a 3-D display according to Embodiment 3. 
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Fig 28 shows a selection criteria with respect to a characterisiic point which is Introduced in Embodinent 3 
Fig 29 shows a corresponding relationship of an original view*;ader image and one re-created so as to be seen 
from a changed viewpoint. 

Fig. 30 shows a corresponding relattonshtp of an original vlewnder image and one re-created so as to be seen 

fnam a changed viewpoint. 

Fig 31 shows an image with h part expanded. 

Fig. 32 shows an image wilh a house separaled from the image in Fig. 2^, 

Fig. 33 shows a structure of a stereoscopic image display apparatus in Embodiment 8. 

DESCRIPTION OF THE PREFERRED EMBODIMENT S 

[0041] Preferred embodiments of tho present embodiment will next be descr bed with reference to the accompanying 
drawings. In Embodiments 1 to 4, an appa-atus outputs as a final imago an image for a 3-D display (a pseudo stere- 
oscopic imago), while in Embodiments 5 to 7, It outputs an image for a 2-D display (an ordinary 2-D image) 
[0042] in Embodiments 1 and 2 the apparatus Initially n^eives a viewfinder image shot on a monocular camera 
while in Embodiment 3 it receives a viewfinder innage shot on a multi-eye camera ja stereo image). Embodiments 1 
2, and 3 correspond to Embodiments 5. 6 and 7, respectively, except that the fomier outputs an image for a 3-0 display 
and the latter for a 2-D display. Embodiment 8 relates to a displaying method, in which unique conditions for a display 
apparatus are considered when displaying a pseudo stereoscopic image. 

Embodiment 1 . 

[0043] Fig. 4 shows main stages for the creation of an image for a 3-D diaolay accoftiing to Embodiment 1 Up to 
the third stage, the content of the method for calculating depth infonnatlon according to the present invention will 
become apparent. 

[0044] In Embod«nent 1 , an image for a 3-D display is created based on an image for a 2-D display through Stages 
1 to 3 for analyzing an Image for a 2-D display, and Stage 4 for creating an image for a 3-D display. Respective stages 
will next be outlined. <• j r a 

[Stage 1] Extraction of 2-D Motion Information 



[0045] Information about the movement of an object shown in a viewfinder image is fitst extracted The motion in- 
fomiation is 2-D at this stage. That is, coordinates are overlaid onto a display screen, so that the movement of the 
object on the screen will be expressed by means of 2-D coordinates. 

[0046] In order to understand the movement of the objecL a corresponding elalionship between viewfinder Images 
is detected. A viewffinder image at Time t is designated as a reference frame {hereinafter referred to as "Frame f) 
while a viewfinder image at Time t' is designated as an object frame (hereinafter referred to as "Frame f") In Frame 
t, a plurality of representative points are pre-provided, so that con-esponding points of the representative points are 
traced in Frame V. Frames t and f constitute different-time frames w th each ether, though they are not necessarily to 
be adjacent in terms of frame sequence. Stage 1 Is characterized by the fact that 2-D information can be extracted 
from not only a horizontal movement of an object but also from movements in any direction . Hereinafter in this speci- 
fication, (t) and (f) are defined as time, and a frame Is defined as a unit constituting a viewfinder image in general but 
is not limited to a particular frame of a television picture receiver, which comprises 525 scanning lines, or a screen of 
a personal computer, which comprises 640 x 480 pixels or the like. Alternatively, representative points may be provided 
m not only Frame t but also both Frames t and t'. 

[Stage 2J Calculation of 3-D Motion Information 

[0047] After identifying the 2-D movement of the object. Information about sn actual 3-D movement of the object is 
calculated as 3-D motion infomnation. The 3-D motion is expressed by six parameters; three for translation and three 
for rotation. This calculation is made based on a plurality of pairs of representative and corresponding points. 

[Stage 3] Acquisition of Depth Infonnation 

[0048] Identification of the actua' movement of the object would define a relative positional relationship between the 
objects at different times, Further, identification of this relationship could provide depth Information of the obiect or its 
respective parts (hereinafter referred to respective image parts). 
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[Stage 4] Creation of Image 

[0049] A parallax is determined Dased cn the depm nfortintron sc as lo create ngiit and left images. The parallax 
IS determined such that a closer oDie-ot n >s n mcjo p^r^ c ' naqe pans should have a drtfarent 

5 depth, right and left images should be if-ea uc t^at ti, ceUivp r? c pars of each image have a different 
parallax. It is to be clearly understood that he (ollowi' .g iacis are Jiilsient Irom each other and should not be confused: 
thai Is, the fact that the motion information can be ex, .factec: i ojr. a inovenienl in any ttireclion at Stage 1 . and tho fact 
that the direction in which a parallax is provided is limited to a hori£ontal direction at Stage 4 due to the horizontal 
locations of both eyes viewing the object. 

'0 [0050} Respective stages in Embodimeritl have been outlined above. In the following they will be further described 
in detail. 

[Stag© 1 ] Extraction of 2-D Motion Infonnation 

15 [(K)51 ] Fig. 5 is a flowchart for detection of a corresponding i elaEionship between viewfinder image frames, respective 
steps of which will next be described one oy one 

(Step 10) Providing a Representative Point in Frame t 

^ [0052] As shown in Fig. 6, representative points a(e provided in a Reference Frame t. In Fig. 6. Frame t is divided 
into every 8x8 pixels by overlaying it with a grid, and representative points are provided at every crossing point of the 
horizontal and perpendicular lines of the grid. The representative point of the i-th from left and the j-th from the top is 
expressed as R(i,j); a corresponding point of Pt(i,i) at lime f is expressed as Pt'(i.j). The x and y coordinates of Pt(i,j) 
are expressed, if required, as R(l,j)x and ^(i,j)y. respectively. 

2s [0053] A representative point may be provided not only at a crossing point but at also any desired points. As an 
extreme case, all pixels may be individually designated as indepenoent representative points. 

(Step 11) Setting a Corresponding Point Candidate Area 

30 [0054] Taking an example of Pt(6.4) in Fig. 6 , an area which may possibly include Pt'(6,4) is pre-determined based 
on the assumption that R'(6, 4) be positicned in the vicinity of Pt(6, 4) except for a drastic movement of a viewfinder 
image so as to exceed a predetermined limitation, 'n Embodiment 1, n order to reduce calculation for a positional 
detection, Pt'{6,4) is assumed as existing in an area of 1 00 x 60 pixels in the vlcmlty of Pt(6,4). 

3s step 1 1 can also be modified as follows; 

[0055] 

1, When a viewfinder image moves elatively arasrioaiiy ^vo fiames adjacent ii tenns of frame sequence are 
■*o dctormined as Frames t and t' so as to minimi,;c tho extent of a change in the position of roprcsontaiive point 

between Frames t and t', and thus the risl< of displacing the corresponding point from the assumed area. Of course, 
it is possible to assume the whole image area as a corresponding point candidate area. The risk of displacing a 
corresponding point from the assumed area due to a large movement of the viewfinder Image Is thus reduced, 
although the volume of calculation is -esultanlly increased. 

45 

2, In the above, a corresponding po tn ca-naidate area - a.^ 3Gen determined bas3d on a simple assumption that 
R'(6.4) be located in the vicinity of 6,4). However, when the movement of Pt(6 4) is traced among a plurality of 
frames, a corresponding point candidate area can be determined on the exten.sion of the movement trail This 
method is particularly advantageous in limiting such an area in the case of a viawfinder image with a relatively 

50 constant movement. 

(S 12) Calculation of Non-Similarity ir^ s C orresponcing Poi^! .".incidate Area 

[0056] The position of a corresponding point is suecif ea n "le corrosponoing pomt candidate area. !n this case, a 
55 problem arises when the viewfinder image nroves considerably slowly, contrary to Step 1 1 . That is, when the viewfinder 
image moves only by a small extent, it is aifficult to extract motion information ano thus the risk of a severer error being 
included in information is increased. 

[0057] In order to prevent such a problem. Time f is pte-selected such that Frames, t and t are set apart from each 
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other by some extent, in other words, after conducting statistical analysis to the extent of changes of respective image 
parts, Tme f is selected such that the magnitude of changes or the variance of the extent of changes exceeds a 
predetermined value. Attematwely, Time !' may be selected such that the total sum of or the variance of, the movements 
of more than a predetermined nur^ber of charactenstic points (descnbed later) exceeds a predeleimined value If such 
Time f that meets the above conditions is rot found, the creation of an Image for a 3-D display (or the calculation of 
depth informatton) is discontinued and. instead, an input viewfinder image will be output intact or all image parts of the 
viewflnder image are displayed as if having a uniform depth. 

[0058] in this step, in order to determine the position of a corresponding point, non-similarity betv»een Frames t and 
t IS computed by block matching method That is. the total sum of squared differences of gray-scale levels (non- 
similanty) is computed between one block Having a certain point as its center in the corresponding point candidate 
area, and another block including the representing point, so as to detect a certain point providing the minimum sum, 
which is then determined as a computed corresponding point. 

[0059J Fig, 7 shows block matching. In Embodimant 1 , nine pixels constitute one btock with the central pixel as a 
representative point of the block. 

[0060J Block 1 is provided on Frame t, including R(i.j), white Block 2 is provided on Frame f, including Pf(i,J), that 
IS a tentative candidate for a corresponding point. With a pixel value of a pixel(x,y) at Time t designated as lt(x y) the 
non-stmilarity (hereinafter referred to as E1 ) is generally obtaiied from following Equation 1 . 

E1= II{lt(Pt(i,j)x+u,Pt(i,.i)yfv)-lf(R'(ij)x+u.R'(i,j)y+v))^ (Equation 1) 

wherein two S's relate to u and v. Since u and v respectively take the values of 



for a tentative Pt'(i,j), a squared difference of gray-scale level can be obtained with respect to the nine pixels in total 
Then, while gradually changing the position of Pf (i,j) within the candidate area, a point with the minimum E1 value is 
determined as a corresponding point. 

[00611 Fig. 8 is a conceptual model having the value of El in a perpendicular direction for every Pf (l,j) In this model 
Point Q IS determined as a corresponding point, since it snows a steep peak in non-similarity. In this way, corresponding 
points of all representative points are determined, 
[0062] Step 1 2 can also bs modified as follows. 

1 . In the above, a squared difference of gray-scale level has been calculated, as a non-similarity from a gary-scale 
image. Though, in a color image, the non-similarity may be the total sum of squared difference of gray-scale levels 
m red, green and blue, that is E1r + E1q + EIq, Alternatively, the density of other color spaces, such as an HVC 
density, may be employed, orthe total sum of residual differences may be employed in place of a squared difference 
of gray-scale level. 



2 111 this step, nine pixels constitute one block, though it is preferable that one block is defined including a relatively 
large number of pixels. For example, with a screen having a high resolution, such as that of in a personal computer, 
a work station or the like, experiments have shown that a good result was obtained in case of a block includind 
around 16 X 16 pixels " 

(813) Determination of an Initial Position o? a Corresponding Point 

[0063] Up to Step 1 2, a tentath/e con-esponding point has been determined, though it may not be positioned con-ectly. 
Corresponding points relating to borders or edges of an object may have been determined with satisfactotv accuracy 
though It should be understood that points relating to less characterstic image parts may have been detemiined with 
considerable errors. Such a problam is like^ to arise m a case where the value of El does not show a definite peak 
m Fig. 8. or the like. Fig. 9 shows a relationship between a representative point and its corresponding point the cor- 
responding point being determined up to Step 12. Apparently, although corresponding points relating to character^tic 
parts, such as a house and a tree, and especially their outlines, are positioned with satisfactory accuracy, points relating 
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to the sky or the ground are positioned witn considerable arrows 

[00641 In Step 13 and subsequent Step 1 4 tharefoie, such inaccuraiety poslTroned corresponding points are adjusted 
so as to be at a correct position. In Step 13, me corcept ot nhuai positton is introduced, so ttiatthe initial position 
ot each of the corresponding points is actua ly deternnec m nis ?Top m subsequent Step 14, the positional accuracy 
5 is improved through repeated calculations 

[0065] The initial position Is determined, following eithei way stated be ow 

1 , All corresponding points which have been determined up to Step 12 are equally processed in Step 13. 

Positions where ail correspondinc points are nov.- located are regarded as their initial position tor the subse- 
<o quent processing. 

2. Corresponding Points arc Processed Differcr iy 

As for corresponding points whose positions may riavs been determined with satisfactory accuracy (hereinafter 
rofen-ed to as a characteristic point), positions wnere they are now located are regarded as their initial positions 
'5 On the other hand, as for other corresponding paints (hereinafter referred to as a non-characteristic point) their 

initial positions will be determined based on those of the cnaracteristic points. The corresponding points mentioned 
below can be candidates for a characteristic point, though corresponding points of the following (1 ) to (3) are likely 
to coincide. In this specification, representative points of ihe corresponding points as a characteristk: point are 
also referred to as a characteristic point. 

(1) A corresponding point having a definite oeak m the \'alue of El in Step V^. 

Generally, such corresponding points are quite Hkeiy to have been positioned with a high positional accu- 
racy. 

(2) A corresponding point located in art area including many orthogonal edge components. 
Corresponding points included in areas around eoges of buildings are quite likely to have been correctly 

positioned. 

(3) A corresponding point whose position varies steadily from Frame t to f ar d further. 

30 

The steadiness may be understood as the consistency of a motion vector. Tnerefore, a corresponding point 
moving in a consistent moving direction by a consistent distance as a frame proceeds frames t to t', should further 
be selected as a characteristic point. Concrete"/ speaking, a corresponding point to be selected should have a 
motion vector whose variance Is lower than a predetermined value, because such a corresponding point must 
35 have been traced precisely among respective fiames, and thus having been Judged as having a correct corre- 

sponding relationship with its representative point. However whan the camera has moved iiregularty. the influence 
thereof must be considered in the judgement. 

[0066] When a characteristk: point is detennined, .ts position is used as an initial position, while the initial position 
•« of a non-charactoristic point will bo interpolated by using neighboring chaiacteristic points. In olhor words, since the 
positional accuracy of a non-characteristic point detennined up to Step 12 is low, their initial positions should be de- 
termined geometrically based on the neighooring characteristic points with high positional accuracy. Of course, the 
method of Step 12 can be utilized in findirg a characteristic poirt described in the above (3). 
[0067] In addition to the above-mentioned methods based on the selection of a characteristic point, the initial position 
^ of a corresponding position may be deten-nined by a dynamic programming method. 

(SI 4) Improvement Process for the Positi-jn ot a CoiTesponuirg Point 

[0068] An equation is introduced for eva.uatinp po.3,tional accootaDiiity of a 

50 relative positional acceptability through Iterative calculations with the equation. That is, in addition tc 

Step 12, another equation is introduced for evaluating accepiabiHty of a relative positional relationship between corre- 
sponding points. The evaluation results derived from both ir.g equations are combined to improve the posiiional 
accuracy, 

[0069] Referring to Fig. 10 the principle of reialivG positional evaluation will be described. Fig, 1 0 shows correspond- 
ss ing points. Taking Pt'(l,j) as a center, the following four corresponding pomts are located adjacent thereto- 



ding points, so as to improve 
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[0070] It IS reasonably assumed that R'(i j) is located around the center of gravity of these four points. This assump- 
tion fs based on the expenence mat, even when respeciive image parts n"ove, their relative positional relationship is 
substantially Tiaintained, This experience can be mathomatcaily e.xplained as being equal to a situation where a quad- 
rate differential of Pfti,]), which is a function of i and j, is substantially zero. 

[OOn J Therefore, with the center of gravity of the four points being expres.«od as (Sr(i,j)x, Sr(l,j)y), Equation 2 

E3HPf(i,j)x-Sf(. j)x;V,;Pt'(i.i)y-Sf(i.j)y}^ (Equation 2) 

is obtained for evaluating relative positional acceptability With consideration of Equation 2 only, a corresponding point 
will be most favorably positioned with the minimum E2 value. In other words, relative positional acceptability of images 
IS evaluated using thefucntion of distance between neighboring image parts 

[00721 in this step, evaluation resuits derived from Equations 1 and 2 are combined with an appropriate coupling 
factor k. Therefore, a final evaluation equation E can bs expressed as Equation 3 r k » 



E=t1/N+kE2 (Equations) 

wherein N is the number of pixels included in one block, which has been determined for block matching In other words 
for the »T.provement of the relative positional acceptability, E is first computed with respect to all of the corresponding 
points Then, after adding all E's into IE, the respective corresponding positions are moved gradually so as to minimize 
he value of IE. This computation is repeated until either the value of IE converges or the computation is repeated up 
to a predetemiined number of iterations. That is, concretely speaking, any of the following methods is practiced while 
moving respective con-esponding points. 

(1) A Method Using an Euler-Lagrange Differential Equation 

[0073] When an Euler-Lagrange differential equation expresses LB taking an extremum (a relative minimum In this 
case), a corresponding point is obtained by solving this Euler-Lagrange differential equation. This Is a known method 
According to this method, a direction in which a corresponding point Is to be moved for improvement from its Initial 
position IS determined based on both gradient in respective blocks including a representative point and differential 
behveen corresponding blocks, so that the corresponding point is gradually moved in the direction from its initial point 
until reaching a final solution. 

(2) Fixed Searching Method 

[0074] in a corresponding point candidate area a point is searched where the value of E of a corresponding point 
to be improved becomes minimized, and then newly set as a corresponding point. The fixed searching method is 
characlerislic in that the search is conducted for one corresponding point, while others are kept fixed. The above 
mentioned process is repeated with respect to all con^sponding points. 

(3) Hybrid Method 

[0075] According to the method (1 ), it is possible to position a corresponding point with an accuracy theoretically in 
units of less than a pixel, while according to the method (2), with an accuracy in units of a pixel. Therefore, it is possible 
to utilize both methods, that is. first applying the method (2) to obtain a corresponding relationship with an accuracy 
m units of a pixel, and then the method (1 ) to enhance the accuracy 

[0076] Experiments have shown that the method (2) provides a favorable .solution in a shorter period of time than 
the method (1 ) used to obtain the same level of accuracy 

[0077] Fig. 11 shows the result of improvement processing acco.-ding to th^s step, which has been conducted with 
respect to candidates for corresponding points shown in Fig. 9. Experiments have shown that a favorable result was 
obtained in a color image, with tho coupling factor k at around 5 to 200. Figs. 9 and 11 show model results though 
actual experiments have proved that improvement close to the model results was realized 

[0078] This step is characteristic in that 2-D motion in'crnat on c.n be extracted from the movement of an obiect in 
any direction. This is an advantage achieved through u^ac-stancm;- o' the movement of an object by introducirlg the 
concept of representative and corresponding points. This advan-.age makes the present invention applicable over a 
wider range, compared to a prior art, in which a time difference has been determ ned through the detection of a horizontal 
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[0079] Step 14 can be also modified as iollows: 

1 . In obtaining E2, a center of grav ty of eighi oonis -iriy re delerrmred, the eight points including four points 
diagonally located from the center, ;hat ,s Pt[i , in Fig iO as well as tne four rsspectively located upward of, 
downward of. to the lell of and to Ifie right of ihe cenier Preferably, the optimum combination o( 'he points is 
determined experimenially. as it depends on the kind of 2-D image to be processed. 

2. Evaluation by Equation 3 should beyin with a corrospondirg pcini whose evaluation result of Equation E2 is not 
favorable, because a drastic improvement to sioh a co-responding oomi at an e.arly stage is preferable, as it is 
generally considered to have a ;argc c rro- 

3. For the improvement of the positional accuracy, geoiriatiical .pforrration shoulc be utilized. As tor a plurality of 
representative points forming an area with a geometi ica; reature such as a straight line in Frame t, positions of 
their correspording points should be corrected ?o as to also form the £.amo qoornetncal feature. This correction 
is made for reasons that a part which seems to ne a iine in a viewftnder image is quite likely to form a line in the 
actual 3-0 world as well and a line in the 3-D world ,= t-o jid ^orm a line in Frame t' as well Since the depth of an 
image varies consistently along a line an-dci^r , ul r-ear variation can be visually recognized with ease, 
the correction by the above mentioned niemod y.ili acnieve .a significant improvement. Without such an improve- 
ment, the final image may include irreguianty in depth aiong a :ine. thus possibly resulting In an unnatura: 3-D 
display. As alternative geometncal information, edges of an inage area can be used. 

4. Further, corresponding points are obtained wth respect to other frames as well. In this stage, corresponding 
points are only obtained in Frame t' with respect to Frame t. though it Is possible to obtain corresponding points 
in a third frame, or Frame t". so as to obtain averaged movements of respective image parts. This method is not 
for improving a relative positional accuracy of the corresponding points in Frame t'. but rather for statistically de- 
temiining the movements of respective image parts, oased on the respective positions of corresponding points, 
which have been provided in many fr&mes. and :he respective times when Ihe respective frames are shot. 

5. When an insufficient number of characteristic points sre established, tne ongoing process Is discontinued be- 
cause it Is quite unlikely that an accurate corresponding rel&tlonship will be obtained. 

[Stage 2] Calculation of 3-D Motion Infornriatlon 

[0080] In Stage 1 , the 2-D movements of respective image pa^s on a screen have been identified. In Stage 2, 3-D 
movements thereof are calculated based on the identified 2-f) inrormation That is, sinse the 2-D movement in a view- 
finder image is a projection of the actual R-D moven'ent of an ooject onto a plane the onginal 3-D movement of the 
object Is calculated based on the positional relationship between representative and corresponding points in a view- 
finder image. 

[0081] Movements of an object in the ;i-D world can be gc -orally described as a combination of translation and 
rotation movements. In the following, a method lor calculatinc; a movement comprising translation movements oniy will 
be described first, followed by an example of a generalized method. 

1 . Translation Movements Only 

[0082] Fig. 12 shows a corresponding elationsnip be:weor ihe novement o; Point P on a screen and :ts actual 
movement in a 3-D space. In Fig. 12. the2-Dcoordir'atos arc expiassed with a capital letter, while the 3-D coordinates 
are expressed with a small letter, in which x and y axes are provided on the surface of the screen, while tne ? axis is 
in the depth direction. The distance from tne viewpoint and ne screen is set as 1 , 

[0083] As shown in Fig. 12, P(X,Y) moves to P'(X- Y') n the 2-D screen, while S{x.y,7) simultaneously moves to S 

(x', y', z') in the 3-D space. 

[0084] When the following equation is held 



since the screen is placed a distance of 1 from the vtewer. X. x and y' car be expressed as follows: 
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By solving the above, the follov»ing is Introduced. 



Y'=(Y2+by(z+c) 
[0085J Therefore, with z eliminated, Equation 4 is obtained. 

(a-X'c) (Y'-Y) = (b-Y'c) (X'-X) (Equation 4) 

[0086] Since Equation 4 is expressed In terms of movements on the screen, it is possible to obtain unknown values 
of (a), (b), and (c) according to the Infomiation obtained In Stage 1. However, although. In an actual situation where 
an object that is l< times larger moves at a speed k times higher to a place away k times further the value of k (a scale 
factor) cannot be determined, the ratio of values of (a), (b) and (c) to one another can be solely obtained. Mathematically 
speaking, even if three pairs of {X,Y) and (x',y') are given, since the rank of a coefficient matrix of this smurtaneous 
equation is as low as two, (a), (b), and (c) cannot be determined as real values but only as relative values Therefore 
in this stage, the value of (c) Is nomnallzed to one, so as to express the values of (a) and (b) as a ratio against (c) 
because a ratio Is suffbiently usable in the subsequent processing. 

[0087] An alternative solution with respect to translation movements is as follows. An error (e) is defined from Eaua- 
tion 4, as Equation 5. ^ 

e={(a-X'c)(Y'-Y)-(b-Y'c)(X'.X)}^ 

e={(Y'-Y)a-(X'-X)b-(XY'-X'Y)c))2 (Equation 5) 

Then , the total sum Ze of all (e)'s regarding all con^spondlng relationships between representative and corresponding 
points IS calculated so that the respective values of (a), (b) and (c) are obtained from Equations 6 to 8 so as to minimize 
the value of le. 

d(X6)/da=0 (Equation 6) 

d(le)/db=0 (Equation 7) 

d{le)/dc=0 (Equation 8) 

[0088] More concretely speaking. Equations 6 to 8 are respectively developed into Equations 9 to 11 . 

aI(Y'-Y)'-b2;(X'-X)(y-Y)-cX(Y'-Y)(XY'-X'Y)=0 (Equation 9) 

-aI(X'-X)(Y'-y)^-b5:(X'-Xj^cZ(X'-XKXY'-X Y)=0 (Equation 10) 
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-av(Y'-Y)(Xr"-X YHbI(X'-Xl(XY X'Y}+c£(XY'-X'Y)^-.a (Equation 11) 

2, Movements including Rotations 

[0089] Movements including both translation and lotaiion can be expressed by moans of three displacements in x, 
y and z axial directions and three rotation angles, such as a. |i. and 7, each having a 'espective one of the x, y and z 
axes as an axis of retailor*. Rotation angles can be axpresscd by means of an Eulerlan angle or a roll pitch method. 
[0090] The values of the above six var;al)les are the next to be obtainea. Hoiwever. as explained above, since a scale 
factor cannot be delermlned, the ratio of ihe vanablet- 10 one dnoner is sclely obtained, assuming one of the variables 
as one. It is theoretically possible to spec:fy a movemen. m-.er. given five pairs of representative and corresponding 

[0091] However, ii is to be noted that, depending on the selection of the pairs, the content of movements may not 
be specified by means of solution on a linear iransfomnation in some cases. 

However, it Is known that the selection o' eight pairs could prevent sucn cases, grounds of which can be found in 
references, such as "On the Linear Algorithm for Monocular Storoo-Scopy of Moving Objects" by Ooguchi and Al<iba, 
Transactions of Society of Instruments and Control Engineers, vol.26, No. 6, 71 4/720 (1 990). 

[Stage 3) Acquisition of Depth Information 

[0092] Relative extent of the 3-D movements of the respeciivs image pans have been identified in Stage 2. In Stage 

3, depth Information of the respective image parts is t:btained based on the relative exient. In the following description, 
it is assumed that an object stays still, while the camera shooting the ooject moves instead. For this stage, since relative 
movements between an object and a camera Is the target question, this assumption can be made. 

[0093] The movement of a certain part -n a viewflnder image is expressed by means of a rotation matrix R and a 
translation vector (a.b.c) as follows: 



(x'.y'.i )=P(x,y.z)+(a,b.c) 

30 

The inverse transformation of this equatio 1 . which is expressea as the following Equation 1 2, is considered to be the 
movement of the camera. 



{x.y,z)=R"''|(x'y'.2-, (a.b.c)) (Equation 12) 

[0094] Referring to Fig. 13, the principha for obtaining 3-D coordinates of Point R based on 3-D movements of a 
camera and 2-D movements of Point P on a screen will be explained. This principal is generally known as one for 
triangulatlon, in which, when viewing the direction of Point p from two separate points, Point P (Point S in Fig. 13) 
actually exists at the crossing point of the lines of sight from the two points. 

[0095] In Fig. 13, it is assumed that a camera is moved as indicated by the arrow from Time t to f according to 
Equation 12. Poinl S is projected at Poinl Pi in Frarr;e I and dl Point Pf in Frame I', Point S being a crossing poinl of 
lines Ll and LI'. 

[0096] Since angles 8t and «', which are fomned by the ciir«:iion in whicn the camera faces and Line U and Lt', 
respectively, are known and the direction in which the camera .-noves and its moving distance have been identif.ed, it 
is possible to obtain 3-D coordinates of Point S. Based on the 3-D coordinates of Point S, the respective image com- 
ponents can bo provided with their depth nformation. 

[0097] It is to be noted that, as described above, due to the normalization of (c) as 1 . the obtained 3-D coordinates 
of Point S are having been expanded or (;ompressed by a uniform ratio. However. s:nce it is uniformly expanded or 
compressed as a whole, the depth infonna;ion retains, a correct relative positional relationship among respective image 
parts. 

[0098] In the above mentioned processing at this stage, it is necessary To consider errors which have been cfiused 
up to Ihe previous stage. In other words, cue to such errors Li-os Lt and Lt' often do not cross each other as a 'esult 
of calculation. To cope with such a problem, a point is provided at the middle of a line connecting points on Lines Lt 
and Lf where the lines are closest to each ather. so mat a z cooiainatc of such a point v;^ill be approximately designated 
as a depth of Point S. This process will next be described using an expression. 

[0099] When the direction vectors of Lines Lt and Lt' are respectively expressed as (u.v.w) and (u',v',w'). both Lines 
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L and L' can be expressed as following Equation 13 using parameters of a and fi (real number). 

Lt : (x.y.7)4.o(u.v,w) 

Lt':.;x'.y £')^P(u-.V.W) (Equation^; 
Therefore, when an error (e) is expressed as the following 

e={(x^.pu)-(x■+au')}^+{(y+pv)-(y'+av')}^+{(^+jJw)-(z•4.aw'))^ 

the values of a and p which minimize the value of (e) are obtained using the expressions: de/da^O and de/dB-0 I 
other words, by solving the equations of 

(U +v +w^)a-(uu'+vv'+ww')p+(x-x')u+-(y-y')v4-.:z-z')w=0 

the values of o; and p are determined, so that the depth of Point S is finally expressed as the following. 

{{z+aw)+(7'+pw')}/2 

Especially in the case that the error (e) is zero, the (z) coordinate of the midpoint coincides with that of the crossing 
point oT Lines U and Lt . 

f """"f " Prospectively projected onto the screen of Frame t, so as 

on mlT.il ihL°frf.f °' '^'^ ^PP™^^'^' Line Lt is projected as one point 

on the screen. whBe Line Lr is one line ,n general. Wrth Line Lf expressed as Equation 13, the (x) and (y) coordinates 
Of the points on the projected Line Lf on the screen are expressed as Equations 14 and 15 by dividing (x) and (y) 
coordinates of the points on Line Lf in the 3-D space by their (z) coordinates, respectively. 

x=f(x'+pu')/(z'+pw') (Equation 14) 



wherein (f) is an actual distance from the viewpoint to the screen of Frame t, which can be set as one Bv eliminatino 
P from Equations 14 and 15, Line Lf after being projected (hereinafter referred to as Li) can be specified as follows 



wherein k=v'z'-wV. m=Wx'-u'z', n=uy-v'x'. The closest point to be detected is a foot of a perpendicular from the rep- 
resentative point Pt to Line Li (hereinafter referred to as Point D), that is, a point where a line drawn from the repre- 
Equation T '° '° ^ """^ '^"^ coordinates of Point D are expressed as foltowing 

X = (m^X-k;vl<mY)/(k^+m^) 
y=(k^-mr,-kmX)/(k^m^) 
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Assuming that the original point on Line Lt in the 3-£) space, wnich corresponds to Point D. s designated as Point E 
(x",y-.z"), PoHit E can be detected by subsiituting Equation 1 6 .nt: Equation 14 to obta n (J, and further substituting the 
obtained value of |J into Equation 13. Since p is expressed as 

p = (xr-fx )/ .ff xw). 

by substituting this expression into Equation 13, the ..:) coordinate of ^oint E, thai is z' is determined as the following. 
7"=;'fw'(x2-fx'y(fj'-k-w') 

This can be used as a depth value of Poin: S. 

[01 01 J When the depth value is negative due to errors caused ir image processing, the computed value is not reliable 
because the negative value me-.ns that Point S exists behind the camera Therefore, the (z) coordinate of Point S 
needs to be obtained m some other way, such as by using representative points close by with a positive value 
[0102] Irrespective of which method is utilised, the oomouted depths of the respective image parts shojid be given 
to the respective representative points as an actua, numerica; value. Fig. 14 shows representative points, each of 
which IS gwen an actual numerical value For exampie, rhe depth of Pt(2.3) and R'(4,3) are respectively 100 and 200 
the latter actually being located twice as fsr away as the forrrier 

[Stage 4] Creation of Image 

[0103] A parallax is determined according to the depth information, which has been acquired in Stage 3 so as to 
create right and left images. In this stage, .3 farther in«ge is !o be provided with a smaller parallax 
[0104] in Fig. 1 5, which is a top view of the whole system including an object and a camera, parallaxes are given 
according to depth information. When Pt(i ,3) and Pt(4.3) of Fig 14 .are provided on a viewfinder image shot by the 
camera under the situation shown in Fig. 15, their actual positions are at St(2,3) and St(4,3), respectively the latter 
being located twice as far away from the camera as the former 

[0105] R and L screens and R and L viewpoints are respectively placed as shown in Fig. 15 the R and L viewpoints 
respectively corresponding to right and left eyes of a viewer Then, St(2.3'i and St(4,3) are projected on each of the R 
and L screens by viewing them from respective R a- d L . ewpomts. This projection is carried out with respect to ail 
representative points until a final image s 'ormea o,n each oi the R and L screens. The final Images can be used as 
right and left images, respectively. By displaying such images on a display of lenticular lens type or the like which is 
disclosed in JP Application Laid-Open No. Hei 3-65943. it is possible to obtain a good stereoscopic Image. ' 
[0106] In this embodiment, the stereoscopic image may be generated for a desired part only which has been sepa- 
rated from the image. Taking as an example a scene where a person is located 5 meters from the camera with moun- 
tains in the background. Image processing on a condition ot 'v^ithin 10 metere in deoth" would make it possible to 
separate the area including only the perso.i from the whoio imace aroa, so that right and left images can be created 
with respect to only the area containing the person, while leaving tne rest blank or pasting the area with the person on 
pre-prepared different images. This stage differs frorr stages up to Stage 3 in -he number of a viewfinder image frame 
to be used. Up to Stage 3, at least two frames are usea in extracting required information, though in Stage 4. one frame 
IS sufficient for the creation of right and left inages Fig. 1 6 shows right and left images, which have been created using 
Frame t as reference, in which the image parts of a tree, a nouse and a pe.'son are respectively located a smaller 
distance from the viewer in this order The image part ot the person, closest to the viewer exhibits the following features: 

1 . having a largest displacement to she :ert in a ^t: s-age 

2. having a largest displacement to tho right in a left image. It is respectively understood that the above (1) is a 
situation where the viewer sees the person from a point wnicn is slightly riqhtward Irom the original viewpoint and 
the above (2) from a point whch Is slightly leftward from the original viewpoint. As a result of these features the 
person is perceived as being a smaller distance f-om, that is closer to, the viewpoint. In Fig. 16, the displacements 



of the respective image parts are indicatec 
person, the house and the tree presen: « 



s in the grid, m which the 



[0107] For image creation based on Frame t the respective divideij parts or a viewfinder image in Fig 1 6 
transfonned. In :his case, it is necessary to se.cct either a linear or a non-linear transformation as follows. 
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1 Non Linear Transformation 

[0108] As IS shown in Fig. 16. some af the divided parts are transformed into a trapezoid. A wkte(y-usect linear 
transformation, such as an affme transfomiation, nowever, cannot be applied to such transformation Therefore In 
order to transform a part with four vertexes into a trapezoid, a non-linear transformation, such as a perspective trans- 
formation, is applied. 

2, Linear Transfomiation 

[0109] In the transformation into a trapezoid, provided that a part with four vertexes is first divided into two parts 
each having three vcrtoxes, a linear transformation can be applied with respect to such a part 
[0110] As a result of a horizontal displacement of respective imago parts through the above mentioned transforma- 
tion, the peripheral edge of the image may become uneven, in Fig 1 6. the bottom parts of right and left images are 
displaced inwardly with respect to each other and accoro.ngly the peripheral edges of the displaced parts become 
crooked. Therefore, by adding pixels to such a recess, (he shape of the image is corrected back into its original shape 
(a rectangle in this example). 

[0111] The depth of an image part that falls on the added pixels is determined while referring to the depth of those 
Close to the pixels, or in other ways. The images on the added pixels can be seen only by one of the eyes which is a 
natural phenomenon and arises in an area close lo a window frame when people look outside through the window It 
IS to be noted that this correction can also be made by deleting redundant pixels, which project outward of the edge 
Alternatively, the peripheral part of the image is uniformly cut off by a certain width With this con-ection irrespective 
of the selection among the above methods, it is possible to maintain a natural display. 

[01 121 In this stage, a parallax Is detemriined according to a depth, though the parallax is preferably further adjusted 
for the following reasons. 



[0113] In the above example, it is not desirable for ever a person closest to the viewer to be given an extremely 
small depth, because an image perceived to be excessively frontward from the screen would cause fatigue on the 
viewer's eyes. According to a report in "Nikkei Electronics' (April 4, 1988, p.211), it Is most desirable for respective 
image parts to be given a depth in a range between 0,2m to 2m, when a display is positioned 50 cm from the viewer. 

2. Personal Preference 

(01 14] Some people prefer a close image lo be displayed much closer and a distant image much farther, while others 
prefer the opposite. 

3. Processing Capacity 

[0115] If all image areas constituting a far background, such as a mountain, are displayed as if having the same 
distance, the volume of data to be processed can be reduced. 

[0116] Because of the foregoing reasons, in this stage, the following functions for transforming a depth or a parallax 
are applied as requested. a f k 

1 Depth Transformation Function 

[0117] A depth is directly subject to either a linear or a ncn-linear transformation. That is, the object of the transfor- 
mation is a depth, and a parallax is resultantly changed. For example, as for a viewfinder Image compnsing imaqe 
parts with depths in the range of la to 10a f(a) being an arbitraiy ^alue). the depths of the respective image parts can 
be un, omily multiplied by ten, so that all depths fall ,n the range of 10a to 100a. This depth transfomiation function Is 
advantageous for a viewfinder image with an excessively small depth as a whole 

[0118] Alternatively, when the depth is in the range of 0 to 100a. the depth may be compressed, for example to the 
range such as 25a to 75a wfth 50a as the origin of transfomiation. As a further alternative, all images having a depth 
of equal to or less than 20a, or equal to or more than 1 000a may be iransfomied so as to have a unifomi 20a or 1000a 
depth, respectively. In this case, however, as a result of the umfomi transformation, areas at the upper and lower 
limitation values, that is 1 000a and 20b, become discontinuous, and thus form an unnatural display in some viewfinder 
images. In order to solve this problem, a non-linear transfomiation is applied .such that images smoothly converge at 
the upper and lower limitation values. In this example, the following transformation should be made- 
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v/herein (z) is the original depth, 20 - 20a a - 1000a - 20a ^ 980a. und ~ = 4, 
2 Parallax Transformation Function 

[01191 A parallax is subject to a linear or a non-linear traris-orniation. That is, after a parallax is calculated based on 
a depth and transformed, the depth is recalculated based on the transfomied parallax. 
10 [0120J Fig, 1 7 shows a non-linear transformation to a parallax in which Point S, an object for the transformation, is 
provided on a central line L and Point B is at the foot of a perpendicular from Viewpoint A to the Line L. The depth of 
Point S is expressed by a sogmoni SB, and the parallax S (striolly speaking, a half of the parallax) is set as shown in 
Fig- 17. 

[0121] Taking as an example a case where the parallax Is reduced to a half. That is. Point S is to be transfomied to 
1S a point which satisfies the following Equation 17, that is Polr.tS'. 



(Equation 17) 

20 The depth of Point S' is expressed with a segment 3'B. A sanes of processes m correction with the transformation 
will be mathematically explained. First, 8 is determined using me depth SB, according to the relationship of 

d=atan(SB) 

25 

S'B Is next determined according to the relationship of 

S'B-tane 

so that S'B will be used as depth informe^tion after the transfoimation Since a far point is transformed to be much 
farther and a close point Is much closer than througr a simple inear transformation, the sense of depth is more effec- 
tively adjusted through this transfonnatiori. Equaiior 1 7 ;xp sss os & sir^pie linear scaling, although a variety of non- 
linear transformations, such as Is described in 1 (Non-Linear Transformalioii). can also be applied to the transformation 
35 of 0'->0. 

[01221 According to Embodiment 1 of the present nvention, an image is newly created based on depth information, 
instead of a combination of existing viewfinder images. Since this creation does not require a horizontal movement, 
which has been a mandatory in a conventional time difference m sthod, the present invention is applicable over a wider 
■ range. In addition, since a method for detecting a corresponding point with respect to a representative point is disclosed 
40 in the present invention, it is possible to automate tne extraction of depth information and creation of an image with 
ease and efficiency. 

Embodiment 2. 

4S [0123] The optimum apparatus for practicing EmDodimenr 1 will 03 described. 
[0124] Fig, 18 shows a hardware struajre for pra::ticing ErbodhTienl 1 

[0125] In Fig, 18, a viewfinder image to be prccossed is supplied via an image input circuit 20, whereupon it is 
converted into a digital signal. The digital viewfinder image ;s stored by a frame memory control circuit 22 in a Tame 
memory 24. Subsequent to the memory 24, a corresponding point detection circuit 26 is provided for reading out a 
so plurality of viewfinder Image frames for detection of corresponding points. In the detection circuit 26, the process at 
Stage 1 of Embodiment 1 is practiced by neans of hardware, in which an MPEG encoder or the like is used for block 
matching. 

[0126] The coordinates of correspondir g points wh.ori h^ve jaer- detectfid in the circuit 26, .are stored in a lorre- 
sponding point coordinate memory 28, so as to be arbitra,'ily reao out oy a movement deteciion circuit 30 in the 
55 movement detection circuit 30, the processes at Stages 2 and 3 of Embodiment 1 are practiced, in which a 3-D relative 
position of the object is calculated based or its translation and .'otatlon mcvemeils. 

[0127] The calculated infonnation about tne 3-0 relative posi'ion is suoplied to an image creation circuit 32. where 
tne original digital viewfinder image is retrieved from the fnsme memory 24 so as to create nght and left images, re- 
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spectively, by giving an appropriate parallax between them. Prior lo the image creation circuit 32 an instruction input 
section 34 is provided for receiving seversl instructions [rem outside. 

[0128] The right and left images, which have beer created in the image creating circuit 32, are converted into an 
analog signal by an image output circuit 36 to be s jppiiec to an unillustratec display. 
[01291 The operation o( the apparatus win ne« be descnoea 

[0130] A camera shoots an object so as to capl.re ,is v^ewHnder :n,aye, Cr, . video equ,pn,em Diays a viewfinder 
image. Such a viewfinde, image is supplied via the viewlinder image input circuit 20. so as to be stored in the frame 
memory 24. For a nomnal 2-D display, the supplied viewfinder mages will be displayed intact, or alternatively the 
viewfinder images stored in the frame memory 24 are sequentially read out therefrom for display. For a 3-D display a 
plurality of frames on a viewfinder image are read out from the frame memory 24, so that depth inforniation of the 
object will be obtained from the readout frames, with the conesponding point detection circuit 26 and the motion de- 
tection circuit 30. Subsequently, the image creation circuit 32 creates right and Ich images accordinq to the deoth 
infonnation. 

[0131] The instruction input section 34 can be structured as follows so as to achieve the following functions. 
1 Structured as a Control Knob 

[0132] The sense of depth on the created image can be adjusted so as to satisfy the petsonal preferences of a user 
by varying the sense of depth through scaling depth with a control knob. The rotation of the knob may be adjusted in 
advance such that the minimized sense of depth will provide a 2-D display. 

2. Structured as a Pointing Device 

[0133] 

(1) The sense of depth is adjustable In units of image parts. For example, when the person in Fig 16 is desired 
to be displayed much closer, a pointing device, such as a mouse, is used to point to the person, and then clicked 
As a result, the image creation circuit 32 transforms the cepth information of the person for use in an enhanced 
display by giving a wider parallax. 

The effect of this adjustment will become more significant if the display area of the selected item is also changed 
together with the change of the sense of depth. Concretely speaking, with the halved sense of depth, the display 
area of the selected item will be expanded four times. 

(2) A viewfinder image seen from a different point can be created. Sines the depth infonnation is available by 
designating a shooting point (a viewpoint) of the object through clicking with a mouse, it is possible to compute 
through a calculation of translation and rotation movements of the respective image parts, the movements to be 
caused accompanying the change of the viewpoint. Therefore, a viewfinder Image to be seen from a different point 
can be created. In Fig. 1 6, for example, a viewfinder image to be seen after the viewpoint is changed In height or 
by advancing or withdrawing the camera, can be re-created. Further, since the depth information of the re-created 
vrewfinder imago can be computed through tho calculation, its 3-D display can be maintained in a good condition 
with the parallax changed according to the newly computed depth information A viewfinder image seen from a 
different point will be further described later in Embodiment 5. 

[0134] In the following, the results of experiments, in which the apparatus of the present invention is mounted in a 

work station, will be described witii reference to the drawings, 

[01 35] Figs. 1 9 to 26 show image creation procedures with the apparatus according to the present invention Each 
of the drawings is a BAV picture on a display which comprises an area includ ng about 640 x 480 pixels 
[0136] Figs. 1 9 and 20 are vievirfinder images m Frames I and t', rospectively exhibiting some movements between 
them due to a positional difference of the camera Fig. 21 shows the same viewfinder image of Fig 19 with a grid 
overlaid and representative points provided Fig. 22 shows the same viewfincer image of Fig. 20, with correspondinq 
points at their initial position, in which the initial position is set at the temporary best point. The temporary best point 
IS obtained through block matching, which has been conducted, beginning with a characteristic point, with respect to 
an area of 1 6 x 1 6 pixels with a representative point at its center 

[0137] Fig, 23 shows improved positions of corresponding points present.nc a signifteant improvement from Fig 22 
fLVoT''t°^ ^ ^ considering a pcsitional relationship between corresponding points 

[0138] Fig, 24 expresses depth infonnation at a gray .evel. where a lighter level represents a smaller depth It can 
be seen from the drawing that depth information has been acquired with considerable accuracy. 
[0139] Figs. 25 and 26 are right and left images, respectively, which are created based on depth information A closer 
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object, a can in this example, Is shown to lave a wiQsr paraliax, and is thus given a larger horizontal cJisplacemsnt. 
[0140] As described above, with the present apparatus, it is possitjte to automatically practice the method of Em- 
bodiment 1 of the present invention. Furthsr. the application ot hardware tor block matching contributes considerably 
to the improvement of processing speed. (;omparea :o ar exscutinq -ime required witn tne application of a software. 
[0141J The present appatatus can be effectively embodied ir^ a ptoduct by attaching an add-on-card having the 
structure as shown in Fig. 18 to a personal computer or a worK station, or p-einstalling a circuit having Ihe structure 
as shown Fig. 18 in a television receiver, a video player or ihe like. Further, by combining the preseni apparatus with 
a camera, it is also possible to shoot an object separately from xs surroundings in oider to capture many viewfinder 
images seen from different points, so as to produce a catalogue containing 3-D pictures of the object. With this way 
of shooting, depth measurement by means of a laser, infrared -ays. or supersonic wuves, which have been conven- 
tionally necessary, are no longer nocossaiy. 

Embodiment 3. 

[01 42] Contrary to Embodiments 1 ano 2, in which a mcnocu ar camera shoots an object, in Embodiment 3, a multi- 
eye camera system is used to capture a s-tereo viewfinder image. The captured stereo viewfinder image Is used for 
the creation of an image for a 3-D display in the following, s method for such an image creation is described mainly 
in view of the difference from Embodiment i 

[0143] Fig. 27 shows mam stages through which an image for a 3-D display Is created. The difference from the 
stages in Fig. 4 of Embodiment 1 is the following. 

1 . At Stage 1 in Embodiment 3, displacement information is extracted instead of the motion Information In Embod- 
iment 1 . 

While different-time frames are processed in Emoodiment 1 , same-time frames are mainly processed here in 
Embodiment 3. Between frames shot at the same time, the movement of an ODject cannot be defined. ~hus, 
information about the object displacement between s jcn frames Is extracted instsad. 

2. Stage 2 in Fig. 4 is unnecessary in Rg. 27 

Fig. 27 does not Include a stage corresponding to Stage 2 in Fig. 4 (Calculation of a 3-D Motion information) 
because the distance behveen cameras is already known as shown in Fig. 13 ar d depth information can be ob- 
tained according to the principle of triiipgulation jsing the distance. 

When inaccuracy could be caused with resaect lo the relative positional relationship between a plurality of 
cameras of a multi-eye camera system, it is des: able to jse selfcalibration to coriect such inaccuracy at Stage 2. 
Methods for selfcalibration are described in references such as "SelfCalibratlon of Stereo Cameras" by Tomita and 
Takahashi, Journal of the Information Processing Society of Japan, Vol. 31 , No 5 (1990) pp.650-659, JP Laid- 
Open No. Hei 2-138671 and JP Laid-Open No. Hei 2-133672. 

Stages 1 to 3 of Embodiment 3 w.ll next be described. 

[Stage 1] Extraction of 2-0 Displacement ■nformatior 

[0144] Inadditiontothesubstitutionof motion information With liisplacementinfonnation, Frames tandt' are replaced 
by Frames 1 and 2, which are respectively shot by Camera 1 and 2 at Time I. In Embodiment 3, it is possible to create 
a final image based on a minimum of only nwo frames, which are shot at the same time, that is, Time t. In other words, 
when shooting using a multi-eye camera, the viewfinder imago c;io;ured may be a still image. Stage 1 is further different 
from Embodiment i as follows. 

(1) In Step 11 of Embodiment 1 (Set ing a CoiTespona:ng Point Candidate Area), the amount of calculation is 
reduced with the appropriate selection of differert-time franes or the iimitation of a corresponding point cancidate 
area, which are conducted based on the intensity or 'he trails of the movement of a viewfinder image. In Embod- 
iments, on the other hand, a different method from that in Embodiment i is employed as described in the following 
to limit a corresponding point candidate area foi- the same purpose. 

it is assumed that a multi-eye camara is posil'oned horizo ital'y, as is usually the case. Y coordinates (a vertical 
coordinated of con-esponding points in frames s-ct bv Cr^r .jras i f the muiii-eye camera system are substantially 
the same as one another. Taking this into cons.dera.io.r s.s well as sirors due to image processing or camera 
installation, a corresponding point candidate area can be ;inii;ed to a horizontally longitudinal band area l^oreover 
it is assumed that Frames 1' and 2' ars shot at -me V and Frames 1 and 2 at Time t, respectively wherein I'=i-1 . 
When a positional difference of the representative points between Frames 1' and 2' is x. it can be predicted that 
corresponding point candidate areas in Frames ! and 2 be set so as to have the same difference of x, or therea- 
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bouts, between each other, h other words, the cor-esoonding Doini candidate areas in Frames 1 and 2 can be 
lifnited to the regions, the difference between whic~ is out x 

(2) Although statistical analysis Is introduced for 8 stow movement in Siep 12 of Embodiment 1 (Calculation of 
non-simiiarity m the area for candidates for a corresponding point), this analysis Is unnecessary In Embodiment 3. 

(3) Similarly to Step 12 in Embodiment 1 , block matching is int. educed in determining positions of corresponding 
points ,n Embodiment 3, However, m Embodiment 3, siased block matching may be more effective than simple 
block matching in some cases, such as when the multi-eye camera system to be used is constituted of cameras 
witn different characteristics, -or example, if Camera 2 tends to produce more bluish Images than Camera 1 , the 
color density of Frame 2 should have its blue components (B) subtracted to a certain extent {that is, a color de- 
flection constant Oq) before undergoing block matching. Without such an adjustment, there is a risk that the moan 
ing of E3 for combing El and E2 may become invalidated. An exampls w,ll be taken where a color density is 
expressed in red, green and blue spaces. In such a case, not only blue (B), but also red (R) and green (G) should 
undergo such an adjustment through subtraction of ccor deflection constants Op, and a^, respectively. Note that 
the biased block matching evaluates the similarity, based on a ,=;quared difference of gray-scale level. This means 
that the similarity car be treated as a distance in the color space, which Is the same metric as is used for relative 
positional acceptability of viewfinder images. Therefore, the similarity and the acceptability can be combined to- 
gether and can be used for the matching evaluation. 

Referring to Fig. 7 and based on Equation 1 , biased block matching will be described using equations. Pt(i,j) 
in Embodiment 1 is denoted as P1 and P2 respectively corresponding to Frames 1 and 2, and lt(i,j) is as II and 
12. Since Equation 1 can be simplified to be expressed as Equation 1 8. Equation 1 8 can be used in nonnal blocl< 
matching with respect to a gaiy-scale image. 

E1=i:E{l1(P1x+u,Fly+v)-i2(=2;<.fU,P2y+v))2 (Equation 18) 

On the other hand, biased block matching is represented by followmq Equation 1 9, which is a modifteation of 
Equation 18. 



For a color image, with a being any one of ur. Uq, and ag, E1 is calculated for all viewfinder images in all RGB 
spaces, so as to obtain the total thereof, that is, E1r + EIq + Elg, which is used in block matching. For simplicity, 
Equation 1 9 can be expressed as Equation 20, with ! 1 and 12 representing 11 (PI x+ u. PI y+v) and l2(P2x+u P2v+v) 
respectively,, v > r /. 

Ei=5:E(|i -I2 a)^ (Equation 20) 

wherein 11 and 11 are functions of u and v, respectively, and a is a constant. 

The optimum value of a is obtained next. Since Cameras 1 and 2 shoot the same object viewfinder images 
captured by both cameras should comprise substantially the same content, except for the displacements of the 
respective Image parts. In other words, the more similar the characteristics of the cameras are the smaller the 
value of El in Equation 20 becomes. Based on this fact. It is known that « should be a value which can minimize 
the value of El . Since Equation 20 can be expressed as Equation 21 . 

E1=ir((l1-l2)'-2a(l1-l2)V} =£Iil1-l2)2-2a2;i(|i-i2)+Tia2 (Equation 21) 

provided that the total number of pixels in a block is N. Equation 21 is further expressed as Equation 22, for SSI = N 

El=IZ(l1-l2)--2aj;i{h-l2)+Na^ (Equation 22) 

Therefore, since 
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is held, the value of E1 Is minimized when Equation ?.c •& held 



Since a can be understood as an average difference value in color densities of respective pixels between two 
subject areas of block matching, a substitution of Equation 23 into Equat;on 22 would lead to Equation 24. 

E1 = li;{M.i2)'^-{lv(|,.|2)(2/M (Equation 24) 

Thorofore, it is concluded that Equation 24 is used for rias.xi block maiching. With the introduction of Equation 
24, if it is assumed that Cameras 1 and 2 shoot exactly the same object, the value of El becomes zero. On the 
other hand, the value of E2 also becomes substantially 2ero . Therefore, it is understood that biased block matching 
is effective in eliminating an initial error caused by the judgement as to the similarity of viewfinder images. After- 
w/ards, the best matching will be searched through the same process as that in Embodiment 1 , 

It is to be noted that a color density other than an RGB density such as an HVC density may also be applied 
without a problem for block matching. Moreove'-, block matching may be carried out based on a color ditferunce, 
that is a residual difference, instead of a squared difference af gray-scale level. When a correction value a. which 
has been determined by Equation 23. exceeds a predeternined value range, a biased block matching may be 
discontinued. It is necessary to provide such a ma);imum iimitation value, without which, sometimes the block 
matching may detect an incorrect con'esponding point because the block including a point at Issue has accidentally 
the similar pattern although it has a quite different color. However, since the color difference caused by camera 
characteristics is generally not very large and therefore is within a predetermined limitation range, an introduction 
of such a limitation value would be useful a,na pract cal. 

With the biased block matching discontinuod, normal b oci< matching may be used to evaluate the similarity 
of viewfinder images. Alternatively, the value derived (ro^i the biased block matching may be used after con-£<;ting 
Image parts only at the upper limited value of a coirectabis range (hereinafter referred to as T), which can be 
computed with the following equation. 

E1-L£(|•|-l2)^-{lv(|t.|2)}2/^J.^^ix^ 
Wherein x=ili;(ll-l2).''Ni-T 

(4) In Step 13 of Embodiment 1 (Determination of an initial Position of a Corresponding Point), a point with a stable 
movement among different-time Frames t, t' is funher selected as acharacterlstk: point. In Embodiments, additional 
criteria are considered for the selection, in Fig. 2S. Frames 1 0 to 12 constitute different-time frames to one another 
shot by Camera 1 , wnilo Frames 20 to 22 conslrtute ditfarsnt-time frames shot by Camera 2. Two frames shown 
side by side in Fig. 28 constitute same-time frames as each other. While directing attention at Point P in respective 
frames, its movement between the different-time frames is expressed with a vector An (n being a natural number), 
It between the same time fram.es with s vecto- Bn. 



[0145] When set as described above, a point which meets the following criterion wii| be selected as a characteristic 

(a) vector Bn is substantially consiste.Ti or moves substantially consistently. 

in addition to the above criterion (a), the fo.lowing criterion (b) may be added, so as to select a point which 
meets the both criteria as a characteristic point. 

(b) vector .An is substantially consistent or moves suDstanlially consistently 

Criterion (b) corresponds to the CDndition Inrroduced in Embodiment 1 As described above, when a shooting 
with a multi-eye camera system, it is possible to obtain depth intormation from same-time frames only For this, it 
is necessary to obtain a correct conesponding reiationship between viewfinder mages. !n obtaining the correct 
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corresponding relationship, information obtainable from different-time frames is encouraged to be aliowed in ad- 
dition. Since it is considered as having been accurately traced, a point which simultaneously meets the above two 
criteria will provide key intonnation in the extraction of 2-D displacement mfonnation. When a camera captures a 
stil! viewfinder image, the known dynamic programming may be applied to obtain corresponding points. 

[Stage 2] Acquisition of Depth Infomiation 

[01 46] Depth information is calculated based on the displacement of respective image parts, which has been obtained 
8t Stage 1 . In multi-eye shooting, where the srtuatwn of Fig 1 3 (S achieved at time t. depth infomiation can be obtained 
by the method disclosed at Stage 3 of Embodiment 1 

[0147] It is to be notod that, since respective cameras of the muiti-cye camera system are situated having a fixed 
relationship with one another, assumed that the relationship among thorn and their magnification rates (a focal distance) 
are known, depth information in a real (absolute) value can be obtained, including a scale factor k. which can not be 
detemiined in Ennbodlment 1 . 

[stage 3] Creation of an Image 

[0148] An image is created through the same process as that at Stage 4 in Embodiment 1 (Creation of an Image). 
[0149] In Embodiment 3, as described in the above, a camera receives a stereo viewfinder image and outputs an 
image for a 3-D display. Therefore, the viewfinder image captured by the camera will be precisely reproduced for output 
in addition to the fact that a desired image can be created through Image processing, including an enhanced display 
as described in Embodiment 1, ' 

Embodiment 4, 

[0150] A technique for creating a viewfinder image seen from a different point, by utilizing a mouse and its clicking 
action IS described in Embodiment 2. In the following, examples will be described, where various viewfinder images 
seen from different points are created for a variety of purposes. 

[0151] As described above, according to the present invention, it Is possible to create a viewfinder image seen from 
a different point without moving a camera In this case, naturally, a viewfinder image seen from a viewpoint which is 
hypothetically located a shorter distance from the actual viewpoint will result h greater accuracy. By utilizing this fact 
the following applications can be achieved. 



When a stereo viewfinder image is available with a double-eye camera system, a viewfinder image with multi- 
viewpoints will be created by hypothetically providing a third camera. In other words, a point at which the hypo- 
thetical third camera is placed is determined so that the third camera is set apart from the other cameras by a 
small space. Then; a viewrfinder image seen from the thus determined point is created. The thus created image 
which is relatively accurate, and two viewfinder images actually captured by the two cameras of the double-oye 
camera system are combined together, so as to create a good viewfinder image with multi-viewpoints Subse- 
quently, additional considerations of depth Infomiation would pemiit the creation of an image for a 3-D display 
which is seen from any of the multi-viewpoints 

2. Creation of a Vtewfinder Image in Siow Motion 

The closest two different-time Frames in temis o* time are designated respectively as Frame t and t" ■ viewpoints 
of Frames t and V are designated respectively as Viewpoint t and f. Although the viewpoint is actually changed 
from Viewpoint tto f from Frame t to t', no viewfinder image between them is available. Therefore, by providing a 
hypothetteal viewpoint between Viewpoints t and f, a viewfinder image seen from a different point, that is, a point 
between Viewpoints t and f in this example, is newly created. A plurality of viewfinder images seen from different 
points are created in this way. Then, a sequential display of such viewfinder images would present a viewfinder 
image in slow motion, which has the following effects. 

a. The movement among respective viewfinder images becomes smooth, instead of an original flickery move- 



b. With a smaller movement of a viewpoint between closer frames in terms of time, the quality of the viewfinder 
image between those frames is not degraded. 

c. Variation in a path where a viewpoint moves from Viewpoint t to t' would provide a different effect on the 
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viewfinder mage in slow motion. 

Additional considerations of deptr. information would peToir the creation of ar, image tor a 3-C display It 
e noted that ttie above mentioned technology :an be appiisd lo sarne-tlrrie frames without problems. 



[0152] Embodiment 5, which is substantially the Sdme as Emoodtment i expect thai it outputs an image for a 2-D 
display, aims to practice the following image processing i.slng depth information 

1 . Change In a Viewpoint 

Accompanying the hypothetical c^iango of a viewpoint, a viewfinder image should also be vaned. According 
to the present invention, when the viewpoint is hypothatieally changed, a viewfinder image seen from the changed 
viewpoint is automatically created while the can-.era iS kept ,ixed 

(5 

2. Partial Expansion or Compression jf an Image 

By utilizing depth information, the most natural ard effective viewfinder Image :s automatically created through 
partial scaling as required. 

20 3. Separation of an Image Area 

For separating a desired image area, it is first necessary to •■ulty recognize respective image areas. For area 
recognition, several methods have been proposed, including a clustering method, but they have had only unsat- 
isfactory results. The present invention permits an accurate area separation in a totally different way from the 
conventional way using depth information 

[0153] Since depth infonmation is obtained througn the same piocess as thut in Embodiment 1 , in the following only 
Stage 4 (Creation of an Image) will be described, as it differ;, from Embodiment 1 . 

[Stage 4] Creation of an Image 

30 

[0154] A desired viewfinder image is created according to the aepth infonnation, which has been obtained at Stage 
3. At stages up to Stage 3, at least two viewfinder image frames have been demanded in extracting necessary infor- 
mation, though it Is possible to create a desired image based on only a single viewfinder image frame at Stage 4, 

■35 {1 ) A Viewfinder Image Seen from a Different Point 

[0155] Figs. 29 and 30 show a corresponding relationship betv/een an original viewfinder image and one re-created 
so as to be seen from a changed viewpoint. Fig, 2S is the c.-igir-al viewfinder image, showing a tree, a house and a 
person, each having a smaller depth in this order Fig 30 is the .lev^inder image created with the assumption that its 
■to viewpoint IS hypothetically moved to a point somewr are .at the lop right of the scene, 

[0156] As is apparent from those drawings, accoroing so the present invention, It is possible to obtain a viewfinder 
image seen from a different point while the camera is fixed, because 3-D infonnation about the respective image parts 
has been known, Including the depth info-mation. from Stage 3. In this example, it was assumed that the viewpoint 
was moved up lo the top right of the scene, althougfi it oar. be understood that the object was moved down to the 
45 bottom left of the scene. The movement to the boliom left can be expressed in the form of translation and rotation 
movements, as described in Stage 3, Bv rsvorseiy -.ilowing ,ne processes at Stages 1 lo 3, it is possible to compute 
a 2-D movement of the object on the scre-n, basse on this ' ypoihetical 3-D movement of the object, so as to create 
the viewfinder image shown in Fig. 30. Since no room is left for arbitrariness in the creation through Stages 1 to 4 the 
thus created viewfinder image is very nattral. 
so [0157] In this stage. It Is preferable to consider and reflect a n-iasKing relationship ir- creating an image, Conctetely 
speaking, in -ig. 30 for example, accompanying a c-iange oi the viewpoirt, the bottom part of the tree becomes ob- 
scured by the roof of the house. Therefore- for creating a nat jra viewfinder i-nage, the bottom part of the tree should 
be covered by the image data of the hoaje act.ai .o-^A re ^Twssing, the creation should be started with an image 
part having a larger depth in order to c^eaie a nat.ra v-e^.rf.naer ^nage. Alternatively che Z-buffer technique v 



widely-used in computer graphics, can be used for tf 



! purpose For obtaining a mask ng relationship through compu- 



a Judgement is first made as to whetner the sigrt vectors di/eaed at the respective image parts from the changed 
viewpoint are overlaid on one another Wfien the sight vectors to Parts A ano B are overlaid on each other and Part 
A is located closer to the viewpoint than Fatt B, it is ^nown tnsi Part A should oe seen as masking Part B An image 
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may be created based on the thus computed information. 
(2) Partial Scaling of an Image 

[0158] In an enhanced display, one of the image display techniques, a closer object may be re-positionedeven closer 
while a dislani object may be I ' 



rn,««. r- in dapth Is emphasUeO between the objecto. 

[01591 For this rmage processing, according to [tie present invention, images are partially changed its scale based 
on depth information. Fig 3i sho«s the same viewncer .mage as Fig 29, except that a part of it Is magnified. As a 
result cf the person be.ng expa-dod, tie person hav rg tre shortest depth among all objects in the drawing, the person 
IS perceived as being much cioser to the viewer. As a , esuit. an effective enhanced display is achieved. In this case 
preferably, a masking relationship is also reflected in the r.owly created vicwfinder image. 

[0160J It is to be noted that, in expanding an area with the shortest depth there is no limitation in the magnifying 
ratio because that area can be expanded without a proolem until it is perceived as having no depth at all However, in 
magnifying an area having around a middle depth in the viewfindor image, that is, the house in Fig. 31 , since it should 
not be perceived as being closer than the person, the magni^ing ratio is accordingly restricted. Violation of such a 
restncton would result In an unnatural viswf inder image. In the expansion accoitling to depth infonnation, as is executed 
m the present invention, it Is possible to make conditions such as that only areas with the shortest depth should be 
magnified, and only ones with the largest depth should be reduced, so as to create a natural and realistic image that 
is, an image in compliance with the laws of nature. 

[0161] In the above, methods have been described for creating a natural image, though unnatural images may some- 
times be demanded, such as when the sense of unnaturalness needs to be emphasized with a larger display of a 
distant part than that of a closer part. Such an unnatural image may be used for games orthe like. In any case, according 
to the present invention, the naturalness or unnaturalness can be freely created as desired. Conventionally a natural 
image may or may not have been created as a result of an accident where the scale of some parts of the image were 
changed. However, according to the present invention, the creation of a natural or an unnatural image is ensured as 
requested. 

[0162] Once a natural image Is created, In order to further carry out the above described process (1 ) or a pnjcess 
(3) to be described later on the natural image created, it is preferable to begin the process by changing the depth of 
the expanded or compressed areas. For example, when an area is doubled in Size, its depth should be halved. Contrary, 
when an area is halved in size, its depth should be doubled. This correction is necessary because the size of an area 
IS inveiseiy proportional to its depth. An image so corrected would ensure a natural image to be produced in the 
subsequent processes. 

[0163] In conducting the image processing of the above (1) and (2), an imaga may be finalized by smoothing uneven 
parts along the edge of the image. For example, when re-creating the viewfinder image in Fig. 20 into that In Fig 30 
It will never happen that all image parts in Fig. 29 correspond to those In Fig. 30 with a one-by-one corresponding 
relationship. Concretely speaking, since the space shown at the top right corner of the image in Fig. 30 may show 
objects not seen at the same region of the image in Fig. 29. Therefore, in a naive creation of a viewfinder image of 
Fig. 30 based on that of Fig. 29, an Image part to be shown at the region is actually broken off in Rg. 30. This break- 
off causes a recess with respect to the ideal edge line of the image. For the same reason, all image parts included in 
Fig. 29 are not shown within the ideal edge of the image in-Fig. 30, while some image parts are projected from the edge 
[0164] In order to solve this problem and to maintain the original screen shape (rectangular in this example) such 
recesses are filled with extra pixels, while such projections are cut off with redundant pixels. The filling should be made 
with pixels having the same color as that of Images in the adjacent region. When the above image processing (2) 
causes similar unevenness along the edge line of the image, a similar amendment would solve the problem. With this 
amendment, the edge line of the image is displayed natural. 

(3) Separation of Image 

[0165] A desired image part is separated to be individually processed. Referring to Fig. 29, it is assumed that the 
person, the house and the tree respectively have a depth of 3m, 10m. and 20m. In order to separate the person only 
a condmon is made such as that "within five meters in depth" before starting the detection and judgement of the depths 
of the respective parts. In order to separate the house the condition may instead be such as "within five to fifteen 

meters in depth." 

[0166] Fig. 32 Is a viewfinder image created with the house separated from Fig. 29. After separating a desired image 
area, the rest may be left blank, or the separated area may be pasted on a different viewfinder image 
[0167] As described above, the present invention provides methods for image recognition and processing as well 
Conventionally, image areas have been separatea manually or by means of a clustering method using colors The 
present invention provides a method for achieving an accurate area recognition in a totally different way from the 
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conventional method, using depth informgtion, 

[0168J As described thus far, the presen! invention discloses a .nethod for image processing using an accurate depth 
intormation. Since the series of processes can be tuiiy automated with software, me present invention can be applicable 
over a wider range 

Embodiment 6 

[0169] An appropriate apparatus in pracucing Embodiment 5 will be described, which is substantially the same as 
that described in Embodiment 2, except that it outputs a single type of image instead of two types of images, that is, 

'0 images for right and left eyes in Embodiment 2 The operation v-.'^! next be described only in connection with structural 
differences from Embodiment 2. A earner.! shoots ai obioc: so as to capture its vievfrnder image, which is supplied 
via the image input circuit 20 to bo stored i t tne frfimc mc-noryi'l A pluraliry of frames on a viowfindcr imago arc road 
out from the frame memory 24 to be processed through the coi responding ooint detection circuit 26 and the motion 
detection circuit 30, for obtaining depth intormation of the object. 

f5 [0170J Subsequently, the image creation circuit 32 creates an image, such as a viewfinder image seen from a different 
point, according to the depth information, ,n this case, witn an instruction supplied via the instruction input section 34, 
various processes will be carried out, including the creation of a viewfinder image seen from a different point, expansion! 
compression or separation, as described .n Embodiment 5. 

^ Ennbodiment 7. 

[01 71 ] A method for creating an image for a 2-D display when -eceiving a stereo viewfinder Image will be described. 
[0172] The difference between Embodiment 5 and 7 is the same as that between Embodiment 1 and 3. According 
to Embodiment 7, It is generally possible to obtain depth information with a high accuracy, and thereby achieve a highly 
25 accurate creation of a viewfinder image seen from a different point as a final Image. 

Embodiment 8. 

[0173] Similarly to Embodiment 1 , a method for displaying a good stereoscopic image, based on information about 
30 a depth and about a 2-D image, will be described. Embodiment 8 is different from Embodiment 1 in that it considers 
conditions unique to a display apparatus, when displaying inag-ss. 

[0174] As disclosed in foregoing JP Publication Na. Si-c r,6 be240. when given depth infom-iation, it is possible to 
create and display a stereo Image, based on a 2-D mage. Tnai is, ihe sender transmits television image signals (2-D 
image signals) appended with depth information. The receiver on the other nand, divides the received image signals 

35 into two groups. Then, the respective image parts of one of the two image signal grouos are given some displacement 
according to the depth infomiatlon, so as to create -ight and left eye images, respectively. The thus created Images 
are displayed on a stereo image display apparatus, ;o achieve reproduction of a stereoscopic image. 
[0175] In this case, it is necessary to conscter the nature oi a parallax. In other word, as has already been discussed 
in the above, since a parallax is based on an flng:, !fir dffreie-ce between sight vectors, the extent of which varies 

■w among display apparatus in different sizes by disDiac ng ever the same numberof pixels, the parallax rosultantty varies 
depending on the size of the display apparatus. Evon assuming ;hat the size is the same, the parallax still varies 
depending on the distance between the display apparatus and the viewer. Therefore, in order to embody the optimum 
3-D effect, the extent of displacement should be oelemnined individually according to the unique conditions of a display 
apparatus. 

'^s [0176] In Embodiment 8, the correction vaiue umot e to each stereo image display apparatus is introduced in addition 
to depth infonnation. Fig. 33 shows a stmciure of a stereo image display apparatus according to Embodiment 8 in 
which a 2-D image and depth infonnation are suppned via en input temiinal 1 00. and the latter is extracted in a depth 
infomnation extraction circuit 102 by a l<,nc wn meth&J. 

[0177] The 2-D image, on the other hand, is divid.?d into two grouos. One is suppl.ed lo a buffer memory 104, and 
50 the other is supplied to a right eye image isplacemont crcuit 1 06 The t u-fo- nemory absorbs a delay caused in the 
displacement circuit 106, A Left eye display panel 108 displays an ima-ae transmitted from Ihe buffer memory^ 104, 
while a right eye display panel 110 displays an image given d;solacerr,ent in the displacement circuit 1 06, 
[0178] This apparatus is characteristic ii tnat the disp.acen-itrt i^rcuil 106 detem^ires the extent of displacement 
with reference to not only depth information but aiso ro pa.ameters unique lo the apparatus, the parameters being pre- 
ss stored in a ROM 112. The ROM 112 stores the optimum corect on value for the apparatus, which is in compliance to 
tne following general rules. 

(1 ) Relating to the size of a display panel: for a smaller jispiay panel, a larger value is stored. 
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(2) Relating to the distance from a display panel to a viewer in an ordinary use: for a smaller distance a smaller 

value IS stared 

[0179] The display circuit 1 06 gives a larger displacement :f a depth is srr-ailer or a correction value is larger the 
correction value being predetermined according to the above men-..ncd , ulev As a result, the optimum stereoscopic 
display is achieved, which reflects the conditions unique to the disuwy apparatus 
[0180] Embodiment 8 may also be applied with the foiiowing lechnieal variations. 

1 . As shown in Fig, 33, a volume 114 may be provided for manually changing the extent of displacement, so that 
a supplementary adjustment or an adjustment according to a personal preference can be performed. 

2. Displacement may be given to both right and left rr^ages 

3. As is described in the foregoing Nikkei Electronics No. 444, a 3-0 effect may be created using a Pulfrich effect. 



5. Con-ectlon values may be stored in two groups, one being changed depending on the screen size, the other 
being changed depending on the distance between the display and the viewer. 

6. In the description above, the value of depth information Is accurately proportional to the distance between the 
shooting position andthe object. Hov«ever. depth information may show the absolute distance between the shooting 
position and the object. A simple case will be taken as an example, where depth information comprises only three 
distinctions: that is, large, medium, and small When depth information indicates "large" or a long distance no 
displacement is made to cause no parallax to the Irrage part. When ■■medium" or a medium instance is indicated 
some displacement is made to cause a parallax to the image part. When "small" or a small distance Is indicated' 
a large displacement Is made to cause a large parallax to the image part. 

[0181] The thus simplified depth infomiation could reduce the volume of transmission data for broadcasting in ad- 
dition to achieve a circuit having a simple structure for a stereo image display apparatus. 



Claims 



A method for calculating the relative depth of an object, comprising; 

a step of providing a plurality of representative points in a reference frame; 

a step of conducting evaluation of similarity of images between an image area including specific points which 
f^^*"^'^"'^ " another frame and a nearby image area including the representative points in the reference 
a step of evaluating relative positional acceptability among the specific points; 

a step of determining the specific points as corresponding points of the representative points when both eval- 
uations provide favorable results; 

a step of conducting a search for a best point where both evaluations provide a most favorable results while 
moving one of the corresponding points with aM the other corresponding points fixed at their current posrtions- 
a step of conducting a positional change of the one rj tno corresponding points to the best point, which was 
found dunng the search; 

3Ct to all of the corresponding 

a step of calculating the depth information according to a positional relationship between the representative 
points and the corresponding points, the corresponding points having been detennined through a series of 

the above steps. 

2. A method for cateulating according to claim 1 , where.n after the search and positional change is conducted for all 
Of the corresponding points, positional accuracy thereof is improved by solving an Euler-Lagrange differential 
equation indicates a condition where a combined value of both of the evaluations is an extremum 
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A method for cateulating according to any one o( claims 1 a-^d 2 wharein 

the evaluation on sinailarity of images is cxinducted by oiassd olock matching where the similarity is cor-ectly 
evaluated to be highest when the blocKs inc'uding ti-.e identical object are tested, regardless of shooting con- 

relalive positional acceptability of rnages :S evaluated usirg ihe function o; distance between 'leighbonng 
image parts, and, 

the results of the above evaluations are treated in teims of distance in the color space and the pixel space 
respectively so that they can be comb.ned together anc: csr be used for the evaluation in determining corre- 
sponding points. 

A method for calculating according to claim 3. vihcie n 

the evaluation of similarity of mages is conducted dv oiased block matching within the limited correction 
range, which has been pre-determined. 

A method for calculating the relative depth of an object, comprising: 

a step of providing a plurality of rspresentatve points in a reference frame; 

a step of determining a plurality cf corresponding points in another frame, so as to correspond to each of the 
representative points; and 

a step of obtaining a positional rfiiationship oetween at least a characteristte point among the representative 
points and its corresponding polr t. 

wherein a point whose position rr'oves steadily among a plurality of frames shot at different times is selected 
as the characteristic point. 

A method for calculating the relative depth of an object comprising; 

a step of providing a plurality of representative points In a reference frame; 

a step of determining a plurality tjf corresponding points In another frame, so as each to correspond to each 
of the representative points; and 

a step of obtaining a positional relationship belweeii dt least a characteristic point among the representative 
points and its corresponding point. 

wherein a point Is selected as the craractoriiiic poir,. w Uise displacement is substantially conslstsnt between 
frames simultaneously shot, and furmer suastantiaiiy consistent or changes substantially consistently in be- 
tween other frames simultaneously shot at a close out diffeient time. 

A method for calculating the relative Jeptn of ar. object, ccrnoi ising: 

a step of providing a plurality of representative points in a reference image; 

a step of determining a plurality of corresponding points in another imago, so as to correspond to each of the 
representative points, respectively; 

a step of obtaining a positional relationship between the representative and corresponding points; and 
a step of calculating depth information according to the positional relationship. 

wherein the calculation of the depth information is discontinuea when less than a predetennined number of 
characteristic points are selected from the representative points. 

A method for calculating according tc- claim 7. wherein 

the representative and corresp'.jnding points a.-e respectively provided in two frames which are inclucJed in 
a 2-D motion image. 

A method tor calculating the relative depth of 
wherein the calculation of the depth 

. A method for calculating reltaive deptn information of a 2-D image, 

wherein when a depth of any point in a ccrtair image is calculated as negative, the depth is interpolated by 
using depth information of points close-by havtng positive depth values. 
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<REPRODUCED RIGHT EYE IMAGE > 



< REPRODUCED LEFT EYE IMAGE) 
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Fig. 3 
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STAGE 1 : EXTRACTION OF 2-0 MOTION 

INFORMATION 



STAGE 2: CALCULATION OF 3-D 
MOTION INFORMATION 



STAGE 3: ACQUISITION OF DEPTH 
INFORMATION 



STAGE 4: CREATION OF IMAGE 



Fig. 4 



EP1 150 254 A2 



(S10) PROVIDING A REPRESENTATIVE POINT 
IN FRAME! 



] 



(S1 1) SEHING A CORRESPONDING POINT 
CANDIDATE AREA 



(S12) CALCULATION OF NON-SIMIURITY IN 

A CORRESPONDING POINT CANDIDATE AREA 



(SI 3) DETERMINATION OF AN INITIAL POSITION 
OF A CORRESPONDING POINT 



(514) IMPROVEMENT PROCESS FOR THE 

POSITION OF A CORRESPONDING POINT 
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